Allow multi indexes search with different query ASTs between indexes

[fmassot]

In #3734, we introduced the multi indexes search feature.

With one limitation: it works only if the resolved query AST is the same on all indexes. And query AST can be different when indexes have different search default values.

There is a use case where searching on indexes with different default values is beneficial. For example, if you want to search on one index containing logs, another containing transactions, etc. Searching for a specific ID in all the indexes is impossible currently (except if you are searching in a JSON field). But if we allow default_fields in the different indexes will make the search experience possible.

Possible Solution

It's easy to keep around one query AST per index and attach the right query AST to the leaf requests.

The main issue comes from how we list the relevant splits for a given query. In the current code base, we call list_splitswith a ListSplitsQuery argument. Everything works well except for the TagAstFilter as it accepts only one tag AST. However, different default values between indexes will lead to different TagAstFilter.

To avoid that, we could fire multiple list_splits for indexes with different TagAstFilter, which will solve our problem.

[guilload]

A pattern that I use a lot in the ingest for supporting multiple indexes/sources in one query is the following:

service Metastore {
  rpc ListSplits(ListSplitsRequest) returns (ListSplitsResponse);
}

message ListSplitsRequest {
  repeated ListSplitsSubrequest subrequests = 1;
}

message ListSplitsSubrequest {
  string index_uid = 1;
  ...
}

Same for responses. That would solve the mult-index list_splits without having to issue multiple requests.