Elasticsearch: fix support for multiple indices

[matyax]

Originally reported in an escalation, this PR fixes support for multiple configured indices, as a comma-separated list.

Which issue(s) does this PR fix?:

Fixes #71092

Special notes for your reviewer:

To verify:

• Provision an Elasticsearch instance with multiple indices
• Configure the data source with a comma-separated list of indices

Expected results: without the fix, no data is returned by the data source; with the code changes, it should work as usual.

[svennergr]

Left a few comments, maybe we should quickly talk about that on Monday's meeting.

[svennergr]

Something we should ideally also do is to dedupe the array. With a "daily" pattern and the index as [logs-]YYYY.MM.DD,[test],[metrics-]YYYY.MM.DD we would correctly end up in multiple [logs-]YYYY.MM.DD and [metrics-]YYYY.MM.DD indices, but test will also be duplicated, i.e. ending in a total of logs-2023.07.05,test,metrics-2023.07.05,logs-2023.07.06,test,metrics-2023.07.06,logs-2023.07.07,test,metrics-2023.07.07 for 3 days.

[svennergr]

would be great if we ensure with a test that multiple indices are joined.

[svennergr]

Even though this seems to work and I might have pointed you in a wrong direction (sorry), I am starting to be a bit unsure if that's the right approach. After reading https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html

index
(Optional, string or array of strings) Data streams, indices, and aliases to search. Supports wildcards (*). Specify multiple targets as an array.

If this parameter is not specified, the <target> request path parameter is used as a fallback.

We should either use an array here or use the correct <target> in the request path parameter. WDYT? We can also discuss with the rest of the squad next week.

[matyax]

Definitely! Let's discuss this next week.

[matyax]

Thanks for the review @svennergr , let's discuss this more in depth.

[ivanahuckova]

I haven't head a detailed look, but I suspect this #67276 broke it. We decided to change comma separated list to array to be consistent with frontend. So maybe we could just revert it to fix the issue. And then we can decide if we want to refactor/change the logic.

[gabor]

i did some testing on how "frontend-mode" handles this and:

1. for things where there is no "interpolation", for example logs, it sends it as a single-string, like:

{
    "index": "logs"
}

this also works if you put more indexes comma-separated, like logs1,logs2, it becomes:

{
    "index": "logs1,logs2"
}

2. if you use the pattern feature, for example daily pattern, and set the index to be [logs-]YYYY.MM.DD, and query for the last 3 days, you get an array-of-strings:

{
    "index": [
        "logs-2023.07.08",
        "logs-2023.07.09",
        "logs-2023.07.10"
    ]
}

3. if you have a pattern, and also comma-separated, it does not work, like, if i extend [2] to be [logs-]YYYY.MM.DD,logs-*, it will generate a broken index-string

[gabor]

after some thinking, i think [3] from above can be considered unsupported. i see two ways to achieve this:

1. make it a single-string: this PR, or just revert the PR @ivanahuckova mentioned above
2. fix the array-of-strings case, that would need a change like this:

 func (ip *staticIndexPattern) GetIndices(timeRange backend.TimeRange) ([]string, error) {
        if ip.indexName != "" {
-               return []string{ip.indexName}, nil
+               return strings.Split(ip.indexName, ","), nil
        } else {
                return []string{}, nil
        }

i think the single-string approach is more robust, WDYT? @ivanahuckova @matyax

[svennergr]

i think the single-string approach is more robust, WDYT? @ivanahuckova @matyax

I am a bit concerned with the statement in the Elastic docs https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html

Specify multiple targets as an array.

The other fix could be to rely on this statement

If this parameter is not specified, the request path parameter is used as a fallback.

and remove the index from the headers and use it from the request path, where multiple indices can be a single string.

[matyax]

Thanks everyone