Speed up exists and missing filters on high-cardinality fields

[jpountz]

The way that the exists filter works is by merging all postings lists. missing just wraps an exists filter into a not filter.

Merging all postings lists can however be very slow on high-cardinality fields. I think there are two ways to fix it:

1. make these filters run on top of field data,
2. or add a new metadata field that we could eg. call _field_names that would index all field names of a document.

Working on field data has the drawback of requiring a lot of stuff to be loaded into memory if the field doesn't have doc values, and the returned filter cannot skip.

I tend to like indexing field names because it would not load anything into memory with a default setup, and the returned filter could skip efficiently since it would be based on a postings list. But unfortunately it could not be used on indices that have been created before we introduce this new metadata field.

[s1monw]

I really like the _field_names approach!

[uboness]

+1 on the _field_names approach... I think we'll find them to be useful for other things as well

[clintongormley]

+1 on _field_names - awesome solution

[dadoonet]

So it's like _all but for fields names?
In the future, could we need to have separate fields names and need to use a "copy_fieldname_to" feature?

Thinking about it loud. May be there is no use case for that...

[clintongormley]

I don't want this to be forgotten, so I've added a v1.3.0 label. No pressure ;)

[jpountz]

@spinscale asked me about the disk footprint of this feature. In general it is very low: its index options are DOCS_ONLY and is only enabled for fields that are indexed or have doc values. Additionally, fields that are contained in most documents will have dense postings lists that typically compress very well.

I did some experiments in 2 extreme cases:

• 5 user-defined fields in addition to the metadata fields (_uid, etc.) that are contained in every document: overhead of ~0.5 bytes per document.
• 100 user-defined fields in the mapping and each document has 5 random fields from these 100 fields: overhead of ~4.7 bytes per document.

This looks very reasonable to me. Even the 2nd case which has very sparse documents takes less than one byte per field per document.