Queries and unmapped fields

[clintongormley]

As discussed in #11806, different queries handle unmapped fields differently. The nested, has_child, has_parent, geo_*, and function_score query throw exceptions, while all of the others match no docs instead.

We need to make the behaviour uniform across queries.

The solution we came up with in FixItFriday is:

• if a field doesn't exist in an index, then replace the query with a FieldNotFound query, which matches no documents
• if explain is turned on, then the field_not_found query will be part of the output
• if the strict_mappings is passed, then the request will throw a field not found exception instead.

This preserves the ease of use of, eg:

GET _search?name=john

without making search requests even more verbose, but makes debugging bad field names easy.

[martijnvg]

Maybe we should then also have a FieldNotFound agg to have similar behaviour for aggs too?

[nik9000]

Maybe we should then also have a FieldNotFound agg to have similar behaviour for aggs too?

From a migration standpoint its always better to return no results than to blow up. I often deploy code that relies on the "if the field isn't there then nothing happens" behavior and create the field over time. Its just more sane that way. So I'm +1 on that.

Last I checked the term and phrase suggester return exceptions if the field doesn't exist and it'd be nicer if they didn't. Not critical. Just nicer.

[rjernst]

Having a "field not found" query seems very odd (just the name). What about returning a MatchNoDocsQuery, but extending that to have a message about why match no docs was chosen (essentially giving a reason, since match no docs alone doesn't make any sense when building a real query)?

[clintongormley]

@rjernst makes sense

[colings86]

We now have #17748 and #17751 which add an ignore_unmapped option to nested, parent/child and geo queries so the user can optionally ignore unmapped field or have an error reported. There is also a missing option that can be used on fieldValueFactor in the FunctionScoreQuery. Will those we can deal with unmapped fields on all queries so I'm going to close this issue. If people disagree we can reopen and discuss

[tragiclifestories]

Will those we can deal with unmapped fields on all queries so I'm going to close this issue.

How do any of these help with a bool query? Use case is a variant on the 'multiple indices' one: I've recently started indexing a field that is used to exclude results with bool/must_not. This works with 2.4, not with 5.x. I don't see how this could be rewritten as any of the queries that support ignore_unmapped

[audriusbugas]

It still does not work with simple terms filter and others. That's a bit sad. Since ES supports queries on multiple indices, we're forced to have exactly same mapping. Are there any other options?

[clintongormley]

terms filter already ignore missing fields so this works just fine. perhaps ask a question in the forum instead