[ES|QL] - Using a wildcard "*" on a none existing index together with a existing index, then the user doesn't get an error from ES

[ninoslavmiskovic]

Elasticsearch Version

8.13

Installed Plugins

No response

Java Version

bundled

OS Version

MacOS - latest

Problem Description

When I use FROM with an index that does not exist together with a wildcard: "*" and an index that does exist, then Elasticsearch does not come back with errors, but Kibana does.

Example:

from kibana_sample_data_logs, noneexistingindex* | limit 10

Adding recording to show-case the error.

Steps to Reproduce

See the recording to re-produce.

Basically you go use this query:

from kibana_sample_data_logs, noneexistingindex* | limit 10

Skaermoptagelse.2024-01-31.kl.17.37.05.mov

,

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[not-napoleon]

The ask here, as I understand it, is for ES|QL to fail the entire query if an index wild card returns zero results. That has the potential to break existing queries, so I've labeled this as a breaking change issue.

[costin]

The behavior can be customized by allowing the user to specify the preferred indices option. (Thanks to @astefan for pointing out).
We should look into exposing this on the request side and/or command itself.

[dej611]

Documenting here all the current behaviours:

• from kibana_sample_data_ecommerce, non-existent-index => ❌ index_not_found_exception - no such index [non-existent-index]
• from kibana_sample_data_ecommerce, non-existent-index* => ✅ data coming back
• from non-existent-index*, kibana_sample_data_ecommerce => ✅ data coming back
• from non-existent-index => ❌ Unknown index [non-existent-index]
• from non-existent-index* => ❌ Unknown index [non-existent-index*]
• from non-existent-index*, other-non-existed-index, kibana_sample_data_ecommerce => ❌ index_not_found_exception - no such index [other-non-existed-index]

I do not mind either fail or make it fail silently as long as one index is available, the problem I see here is a lack of consistency.
One way would be to expose the indices option (a setting mode within FROM?) or align the behaviour, or maybe there's another idea here. As long as we find a single rule for them it would be great.

[timfrietas]

I agree, consistency makes sense here. I would think the default alignment of throwing an error due to a non-existent index makes sense in any combination regardless of expansion, but if that default is both expensive and inconsistent with defaults outside of ES|QL I'm interested in cost of the alternative(s).

[astefan]

Linking here the work in progress: #106636

For the record, this would make a query where the index (without wildcard) name that doesn't exist to not generate an error. For example: FROM employees, nonexistent OPTIONS "ignore_unavailable" = "true" | limit 3 will succeed and return rows from employees.

For a query like FROM nonexistent1, nonexistent2 OPTIONS "ignore_unavailable" = "true" | limit 3 we will still return "unknown index error message". I've created #106805 to investigate the option of returning an empty response in this case.

[astefan]

Closing as fixed with #106636.
This adds options to a from query so that users choose the desired behavior. Also, I don't think this is Breaking anymore, since it doesn't change any defaults.

[bpintea]

Reopening after feature reverting in #108692.