API to list non-empty fields matched by a query DSL

[wylieconlon]

Problem statement

A common interface element in Kibana is a list of "fields". These are displayed using the dotted path notation, such as browser.useragent and browser.useragent.keyword. To help the user with their choices, we want to provide the most accurate list possible. This usually involves starting with the list of field_caps and then filtering based on whether we think the paths returned contain data. To list some specific examples:

• KQL autocompletion across Kibana shows the full list KQL autocomplete should show existing fields and not all fields kibana#24709
• Filter creation across Kibana shows the full list
• Discover shows a list of fields that are present in a sample of 500 documents, and is unable to show multi-mapped fields such as text/keyword
• Lens shows a list of fields. It also loads 500 documents, but shows multi-mapping correctly
• The ML data visualizer starts with many exists aggregations for each field from field_caps
• The Infra apps limit the list based on some custom logic [Infra UI] Limit Metric Explorer fields kibana#43322

As you can see, this is a common request that is handled inconsistently.

Requirements of a solution

• Takes a search query as input
• For each potential path, calculate whether that field exists any docs in the search
• Same response format as field_caps: list of fields as JSON, like { fieldName: { keyword: { aggregatable: true } } }.

Proposed API:

GET my_sample_data-*/_non_empty_fields
{
  query: {
    term: { 'event.dataset': 'system' }
  },
  include_unmapped: false,
  ignore_unavailable: false
}

Response:

{
  "indices": ["my_sample_data-1"],
  "fields": {
    "myField": { 
      "text": {
          "searchable": true,
          "aggregatable": false
      }
    },
    "myField.keyword": { 
      "keyword": {
        "searchable": true,
        "aggregatable": true
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[wylieconlon]

Random sampling could potentially be used to make these queries fast: #53200

[jimczi]

Closing in favor of #56195 which will add index filtering capabilities in the _field_caps API