Default to KQL

[stacey-gammon]

We'd like to switch the default query language to KQL in 7.0.
The option to switch back will still be available and the admin will also be able to disable KQL altogether in the config.

It will give us more data, and we don't want to have to wait till 8.0 to do it in another switch.

We have minimal support for KQL in timelion, TSVB and vega (#28010).
We don't have plans for full support in timelion and vega, but full support is in the works for TSVB (#26006).

Known issues with KQL:

• Wildcard problems (Issue with wildcards, whitespace, and keyword fields in KQL #23001) - it's a bug and we plan to fix after FF
• Anyone please feel free to add more here

cc @elastic/kibana-app @AlonaNadler

[AlonaNadler]

It will be great to have KQL as default, I think there are a few things we should make sure works correctly:

• Saved searches and KQL- how will it work with existing saved searches done using Lucene syntax? since most existing saved searches will be in Lucene, the syntax needs to change to Lucene, will it be only for that search?
• Timelion and Vega don't support KQL KQL/Kuery support for TSVB, timelion and Vega visualizations #17722
• It will not be ideal that search bar works with KQL by default but for filter aggregation users need to use Lucene, we probably won't be able to fix it in time KQL + autocomplete in filter aggregation  #29790
• Exact match concept doesn't exist in KQL - without specifying a field using quotes to get exact match e.g. "error, info"
• Testing - KQL and autocomplete are not new features but changing it to be the default will give them suddenly higher visibility, do we have time for the right testing, the search bar is a major part of Kibana

[Bargs]

Saved searches and KQL- how will it work with existing saved searches done using Lucene syntax? since most existing saved searches will be in Lucene, the syntax needs to change to Lucene, will it be only for that search?

The way this works today is, if you load up a saved search we automatically use the language the search was originally written in. The default only affects new searches.

[AlonaNadler]

• Wildcard problems (Issue with wildcards, whitespace, and keyword fields in KQL #23001) - it's a bug and we plan to fix after FF - @Bargs will look into it
• Open an issue for an indication on which language is currently used (KQL, Lucene) @AlonaNadler
• @lukasolson merged Kuery/KQL: Support not passing an index pattern #28010 - KQL filters Timelion, Vega and TSVB, it doesn't support scripted fields and no autocomplete
• @lukasolson will look into autocomplete in TSVB, the minimal will be to support autocomplete when TSVB is using regular index patterns,
• The advanced setting still says kuery through all the documentation is calling it KQL, will be good to change it to KQL for consistency @timroes will follow up in this
• Default to KQL will be merged in 7.0 beta - @Bargs

open question:

• OSS doesn't have autocomplete

[alexfrancoeur]

I spoke to @AlonaNadler about this yesterday a bit but I thought I'd have to share with this group as well. Just a couple of comments below.

KQL everywhere

• If KQL is the default search experience, as a user I would either expect it to be the default or at least an option everywhere else. Some examples off the cuff include filter buckets in visualizations and TSVB filters. Alona mentioned this above but I'd like to give it my +1, maybe we can open an issue to track?

Performance

• Do we have any concerns about performance of autocomplete in both Cloud and on-prem? I know of at least one customer that has experienced performance issues with typeahead support. I can't seem to find the SDH but can follow up if needed. Given my recent experiences with Cloud / Canvas, I know interesting problems can pop up. Have we done thorough testing for autocomplete on Cloud?

Default experience

• I'm a little worried about making it the default experience for existing clusters. I realize saved searches will still work, etc. I can just imagine someone upgrading their cluster, running a query they're used and being extremely confused. So if we do decided to make this the default for everyone (not just new clusters), I feel like we should be really explicit about messaging, maybe even making this a breaking change for Kibana.

[AlonaNadler]

Is there a way to have Lucene stay the default for OSS users only?

I have a few concern specifically for KQL as default for OSS users:

• OSS users don't have autocomplete capabilities, it will require them to learn a new syntax, Lucene is quite familiar, and is what they are used to. it is likely that without autocomplete users will start typing a query get an error and will need to figure why it doesn't work, it might be a frustrating experience
• Autocomplete is the immediate way to notice we changed from Lucene to KQL, it also helps users build a query in the KQL syntax using the suggestions which OSS users don't have
• Even though there is a way to set Lucene as a default, It might look as we are pushing OSS users to basic in this change and can be perceived a bit aggressive way to do it.

It will be good if we can change Basic+ to KQL while OSS will still use Lucene, @lukasolson @Bargs I know it introduces some complexity, do you think it is possible?

[lukasolson]

Is there a way to have Lucene stay the default for OSS users only?

I'm sure this is a possibility, although I'm not sure we would want this behavior. Seems like our goal should be to push users to use one language for querying Kibana.

In general a lot of the syntax for KQL is similar to Lucene. There are a couple things that don't work which might cause confusion, such as range (bytes:[1 TO 100]). In order to prevent users from getting frustrated we might consider adding special handling to these cases (automatically converting it to the correct syntax, or adding specific error messages to lead them to the correct syntax and documentation). I think this is probably a better long-term approach than keeping Lucene the default.

I think it might help to have some visual indication in the query bar to know which language you're querying in. If we change the default to KQL and don't provide any sort of indication that it's changed (other than error messages when syntax isn't correct), this would probably be unexpected and frustrating.

[Bargs]

Yeah, I think KQL should be the default for everyone or no one at all. For KQL to achieve its goal of abstracting ES details away from user queries, Lucene needs to be phased out as a primary way to write queries. KQL has advantages other than autocomplete, like scripted field support and a more intuitive syntax, and I think we should continue pushing basic enhancements like these into OSS. It might confuse some users, but we have to make the switch sometime. And as Lukas said, there are ways we could reduce that confusion.

[AlonaNadler]

Seems like our goal should be to push users to use one language for querying Kibana.

Eventually, though this is a process rather than a clear cut, especially for OSS users who don't have autocomplete. KQL will get a lot of exposure in 7.0 from default distribution, and we will start getting requests and bugs. As a result, KQL+ autocomplete will get more attention in 7.x and will be ready to default to OSS in 8.0 even without autocomplete. Lucene still have more capability than KQL; autocomplete is one of the main reasons users use KQL (in the future there will be more enhancements to KQL which will drive users to use it regardless of autocomplete)

In order to prevent users from getting frustrated, we might consider adding special handling to these cases (automatically converting it to the correct syntax, or adding specific error messages to lead them to the correct syntax and documentation). I think this is probably a better long-term approach than keeping Lucene the default.

It is a good solution, will you have time to do it for 7.0?

[rayafratkina]

To summarize: we will plan to merge the PR defaulting new searches to KQL (and changes the option to be consistent) asap to make it into 7.0.

We will add a list of issues related to hardening KQL to our bug fix spike. The ones that are bugs will go out in 7.0. The further enhancements will be planned for 7.1 (and beyond). These include

• Wildcard problems (Issue with wildcards, whitespace, and keyword fields in KQL #23001) - bug fix for 7.0 (will probably punt on this for 7.0, see Issue with wildcards, whitespace, and keyword fields in KQL #23001 (comment))
• Add a visual indicator for which language is used (KQL or Lucene) #30277 - 7.0
• Improve error messaging (esp focused on the case where user is using lucene syntax by mistake) (Improve KQL error messaging  #30600) - 7.0
• Update pre-baked saved objects to use KQL instead of Lucene Update pre-baked saved objects to use KQL instead of Lucene #30746 - 7.0
• Autocomplete in TSVB (Add autocomplete to TSVB #30158) - part 1 will go into 7.0 (almost ready). Further enhancements will be done in 7.x
• Telemetry data about language usage, errors and types of queries used
• Making sure KQL is available everywhere (KQL everywhere #30177) - not urgent, moving to the bottom of the list

We will also work on expanding and improving test coverage (including some manual testing if needed). Scenarios to consider

• autocomplete performance in the Cloud
• autocomplete performance in on-prem deployments

Finally, this should be considered a breaking change and communicated as such in documentation and release messaging in order to make sure users are not surprised when switching.

[tylersmalley]

Is this still considered a blocker for 7.0?

[Bargs]

IMO it should no longer be a blocker at this point. The wildcard issue is still open but I’ve made the case in its comment thread that we should punt on it for now and no one has argued otherwise. Telemetry has not been speced out and is not critical for 7.0 IMO. Everything else we wanted for 7.0 is complete.

[lukasolson]

Pretty sure we can close this issue now. The default has been changed to KQL.