Why did tests not pickup errors when types split out from index?

[ranguard]

We need to work out why we didn't get tests failing when we split the types into separate indexes...

These were some of the live errors from the starman_error.log

NOTE: path is does not mean it relatest to the URL request path

/cpan/author/DDB

[error] [Request] ** [http://127.0.0.1:9200]-[400] [illegal_argument_exception] Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can't execute a single index op, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /home/metacpan/carton/metacpan-api/lib/perl5/ElasticSearchX/Model/Document/Set.pm line 156. With vars: {'status_code' => 400,'request' => {'qs' => {},'method' => 'GET','path' => '/cpan/author/PLOBBES','serialize' => 'std','ignore' => [404],'body' => undef},'body' => {'status' => 400,'error' => {'root_cause' => [{'reason' => 'Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can\'t execute a single index op','typed with it [[author_01, favorite_01, cpan_v1_01]], can\'t execute a single index op','root_cause' => [{'type' => 'illegal_argument_exception','reason' => 'Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can\'t execute a single index op'}]}},'request' => {'serialize' => 'std','method' => 'GET','path' => '/cpan/author/DDB','body' => undef,'ignore' => [404],'qs' => {}},'status_code' => 400}

/cpan/author/PLOBBES

[error] [Request] ** [http://127.0.0.1:9200]-[400] [illegal_argument_exception] Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can't execute a single index op, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /home/metacpan/carton/metacpan-api/lib/perl5/ElasticSearchX/Model/Document/Set.pm line 156. With vars: {'status_code' => 400,'request' => {'path' => '/cpan/author/PLOBBES','method' => 'GET','serialize' => 'std','ignore' => [404],'body' => undef,'qs' => {}},'body' => {'error' => {'root_cause' => [{'r/file/VpHSzlTtixV5vXfR7JFgjFQZH8Y'},'status_code' => 400}

/cpan/distribution/ack

[error] [Request] ** [http://127.0.0.1:9200]-[400] [illegal_argument_exception] Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can't execute a single index op, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /home/metacpan/carton/metacpan-api/lib/perl5/ElasticSearchX/Model/Document/Set.pm line 156. With vars: {'request' => {'qs' => {},'path' => '/cpan/distribution/ack','method' => 'GET','serialize' => 'std','ignore' => [404],'body' => undef},'body' => {'status' => 400,'error' => {'root_cause' => [{'reason' => 'Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can\'t execute a single index op','type'uthor_01, favorite_01, cpan_v1_01]], can\'t execute a single index op','type' => 'illegal_argument_exception','root_cause' => [{'reason' => 'Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can\'t execute a single index op','type' => 'illegal_argument_exception'}]},'status' => 400}}

/cpan/file/AAVpqquiW6McJh0nscGrwXhvIlg

[error] [Request] ** [http://127.0.0.1:9200]-[400] [illegal_argument_exception] Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can't execute a single index op, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /home/metacpan/carton/metacpan-api/lib/perl5/ElasticSearchX/Model/Document/Set.pm line 156. With vars: {'request' => {'qs' => {},'body' => undef,'ignore' => [404],'sericause' => [{'type' => 'illegal_argument_exception','reason' => 'Alias [cpan] has more than one indices associated with it [[author_01, favorite_01, cpan_v1_01]], can\'t execute a single index op'}]}},'request' => {'method' => 'GET','path' => '/cpan/file/AAVpqquiW6McJh0nscGrwXhvIlg','serialize' => 'std','ignore' => [404],'body' => undef,'qs' => {}}}

Actual failing URLs examples (from the NGINX logs):

POST /file/_search

GET /author/ABIGAIL
GET /author/ADRIANWIT

GET /distribution/accessors
GET /distribution/ack
GET /distribution/Acme-CPANAuthors-Canadian
GET /distribution/Acme-require-case
GET /distribution/Acme-Time-Constant

GET /module/Acme::%C7%9Dm%C9%94A
GET /module/Algorithm::MCL
GET /module/Algorithm::NaiveBayes
GET /module/Algorithm::TokenBucket
GET /module/TPath::Attributes::Standard
GET /module/Tree::Object::Array

GET /release/TVIGNAUD/urpmi-8.03

GET /source///?raw=1
GET /source/?raw=1

GET /user/profile?access_token=XXXXXXSNIPPEDXXXXXX

GET /v1/author/LLAP

GET /v1/distribution/App-SpeedTest
GET /v1/distribution/App-tkiv
GET /v1/distribution/Config-Perl-V
GET /v1/distribution/Data-Peek

GET /v1/module/Algorithm::Diff
GET /v1/module/Algorithm::Diff::XS
GET /v1/module/Any::URI::Escape
GET /v1/module/App::SpeedTest