Issue with wildcards, whitespace, and keyword fields in KQL

[Bargs]

Kibana version: master

Elasticsearch version: master

A query like machine.os.keyword:win x* returns no results, even though it should match values like "win xp". The underlying query_string query seems to want whitespace to be escaped with a backslash, but the KQL parser doesn't like that. I'm fairly certain this query worked in previous versions so I need to do some additional research and see if something changed in ES master or if this problem does indeed exist in 6.x.

[Bargs]

So after searching my memory and looking back at chat logs from 6 months ago I think it's a bit more clear to me what's going on, but I still have some questions. We use the query_string query under the hood when we see a wildcard. query_string considers the wildcard an operator so it splits machine.os.keyword:win x* into two queries, a term query for win and a wildcard query for x*. Neither of these match any documents so there are no results. I believe that's always been the case so I don't think this is actually a new issue.

What I'm wondering now is, should we treat keyword and text fields differently when wildcards are used? When querying a text field we actually want the splitting behavior of query_string so that normalization will be applied to the token with the wildcard. But when querying a keyword field we don't need normalization and we don't really want to split the value into multiple tokens. When querying a keyword field with a wildcard should we just use a plain wildcard query? Does that make sense to you @jimczi ? Ultimately what we want to do here is give users some way to use wildcards in search terms that have whitespace in them.

[Bargs]

@rayafratkina @timroes @stacey-gammon @AlonaNadler @lukasolson

Lukas and I were just working on this, and unfortunately it's a little more complicated than I thought when I was scanning the issue during our meeting. The solution I proposed above isn't possible today because we don't store information in the index pattern. Adding this information would be a big change since it would require updating a ton of code to use text/keyword as a field type instead of string like we have today, plus there would be breaking change and migration considerations.

We did come up with one possible alternative, which Lukas put up a PR for. In this PR we're generating both a query_string query and a wildcard query for every wildcard query in the actual KQL expression. The query_string will work with text fields and the wildcard will work with keyword fields, and they don't conflict with each other. It has the desired effect, but running two wildcard queries might be a crazy idea for performance reasons. What do you think about that @jimczi? Are there any other alternatives we're missing?

[timroes]

If there is no larger performance implication I think that solution should be good to go for 7.0.

I would suggest we look at that for 7.1 again, but not by splitting the actual string mapping to text/keyword, since I totally agree with Matt here, that this would be an enormous effort and also lots of plugins would break. I would suggest we keep using string as the default mapping, but just add an additional field to the mapping stringType: 'keyword' | 'text' in case it's a string field (we do have additional fields for conflict types, so that shouldn't be an issue). That way we can use that information in a non backwards breaking way, for the most places it actually doesn't matter whether it's text or keyword and for the few places (like this issue) where it matters, we have that information available additionally. But since that is still a bit of a change, I wouldn't want to merge that for 7.0 anymore, so I still suggest we use Lukas PR as a 7.0 solution. Your thoughts?

[Bargs]

Maybe I should ping @elastic/es-search instead of bugging Jim directly :P

[Bargs]

Yeah I agree with that proposition @timroes