_all field disabled starting in ES 6.x

[lukas-vlcek]

As per elastic/elasticsearch#22923 or breaking_6.7_mappings_changes we should expect that _all field (and associated features like include_in_all) will be disabled in ES 6.x. and removed going forward. We should make our index templates are ready for this in advance if possible.

[lukas-vlcek]

In ES 6.x the newly created indices will not support _all.

As of writing in our data model the _all field is disabled for collectd_metrics only but given it is not explicitly disabled it is present in other data model namespaces (consult ES docs).

The question is, do we need this functionality? This is mostly about how customers need to query the data. If they need and rely on anything like the _all field then we will need to build it using the copy_to feature, it is explained here.

I would vote for disabling the _all field as soon as possible but I am not aware what kind of queries users are used to run, ie how much they rely on it. //cc @portante

[portante]

Perhaps we disable it for all ES V5 work we do in the mean time, and leave it as is for existing V2 work?

[lukas-vlcek]

Yes, we can leave it for ES 2.x
I was more curious about if users need functionality like that in which case we will have to add more copy_to stuff into ES 5-6.x
If we do not know at this moment we will just add just feature as we go...

[richm]

Can you give me an example of an operation which can be performed with _all, and how you would perform that same operation without _all?

[portante]

In Kibana, in the search query bar, write: "_all:"find my sentence"", and it will look for the tokens find, my, sentence in every field of documents in the indices it is querying.

If I want to look at every field, I'd have to enumerate all fields in the query string.

[richm]

I suppose /index/_search?q=keyword works the same way.

What does elastic.co tell it's customers who rely on this feature?

If my app uses a search like /index/_search?q=keyword, what should I do instead?

If Elasticsearch is getting rid of this feature, does Kibana 5 support a way to perform multifield searches without having to specify all of them? Kibana future version planned with this?

[lukas-vlcek]

@portante Isn't the _all field used automatically in Kibana when you enter query without any field name? The doc https://www.elastic.co/guide/en/kibana/5.6/search.html suggest this is the case, but it does not provide details about if the _all field is used behind or it takes list of all individual queryable fields on the fly and construct some bool query using those... we would need to check.

Also after quick searching I run into this forum post https://discuss.elastic.co/t/search-over-all-fields-with-all-fields-and-default-field-differ/100217/4 It sounds to me that Kibana may be doing some fancy magic wrt to querying all fields and I am not sure I understand this well at the moment. We probably need to investigate deeper.

[lukas-vlcek]

[Update]: Documentation about ES 6.x mapping changes.

[portante]

@lukas-vlcek, sorry it took so long to respond.

The definition of the _all field is found here in 6.7 [1].

From the first two paragraphs of the above document:

The _all field is a special catch-all field which concatenates the values of all of the other fields into one big string, using space as a delimiter, which is then analyzed and indexed, but not stored. This means that it can be searched, but not retrieved.

The _all field allows you to search for values in documents without knowing which field contains the value. This makes it a useful option when getting started with a new dataset.

It seems you are right about Kibana not providing details of its use, but we can figure out the behavior based on a few other settings in Elasticsearch.

In particular, the index.query.default_field [2].

It seems like Kibana leverages a behavior of Elasticsearch to get the "default field" to search when a field is not specified in a query. Normally, this defaults the _all. We in pbench explicitly set the default field to a different field [3] to avoid inadvertently combing through so many fields in a query.

I'd consider moving to Elasticsearch 7.x at this point.

[1] https://www.elastic.co/guide/en/elasticsearch/reference/6.7/mapping-all-field.html
[2] https://www.elastic.co/guide/en/elasticsearch/reference/6.7/query-dsl-query-string-query.html
[3] https://github.com/distributed-system-analysis/pbench/blob/master/server/lib/settings/run.json#L17

[lukas-vlcek]

We probably need to explicitly remove any use of include_in_all from our mappings when going to >5.x. There is no documentation for this field mapping attribute in ES 6.x and later.

[richm]

@sradco how will removing include_in_all affect ovirt?

[lukas-vlcek]

@sradco @richm see #101