Unmapped fields are taken into account when checking the limit of full text query

[jimczi]

Today we have a limit in full-text query to limit the number of field expansion that we allow. The limit is controllable with the indices.query.bool.max_clause_count setting and we count all fields including unmapped ones. However for queries that target multiple indices it is possible that the list of field to expand the query to become bigger than the max clause setting even if the list of applicable field per index/shard is small. So for instance a query with 2000 fields where 1000 fields belong to one index and the other 1000 belong to the other index would fail because the total is greater than the max clause count. It is also important to note that we currently translate any unmapped field into a MatchNoDocsQuery to ease debugging but we could also completely ignore these unmapped fields if we wanted. I am opening this issue mainly to discuss this behavior and to assess whether it would be beneficial to only count the fields that will make a difference (the mapped ones).

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jimczi]

We discussed this issue in our Search meeting and we've decided to not count unmapped fields when computing the field limit. Today field pattern that don't match anything in the mapping are not taken into account when computing the limit while explicit field are always counted. In order to be consistent concrete fields that don't match in the index mapping should be ignored too.

[zacharymorn]

Hi @jimczi , I’m interested in taking this one, but may need more information to understand the scope of change better.

From your description of the issue & discussion result, we would like to ignore unmapped fields when comparing against the limit. I did some digging and looks like it can be achieved by local changes in the multiple QueryParserHelper.resolveMappingFields where checkForTooManyFields was called.

Also I’m not sure if I follow this

In order to be consistent concrete fields that don’t match in the index mapping should be ignored too.

Is this just rephrasing that unmapped fields should be ignored when comparing against limit?

[zacharymorn]

@jimczi I've created a PR based on my understanding. Please let me know if that looks good to you.