Automatically filter to existing docs with "Metric", when "Terms" is used

[tbragin]

Today, when you construct a chart where you plot a metric overtime, and then you group results by Term, if the metric does not exist in every doc, you still get the terms for all of the docs.

How common is this scenario? Unfortunately, the schema we provide in Metricbeat, makes this use case more common than one might imagine, with a single index across modules.

What can be done to solve this?

(1) As a user, you can manually add an 'exists" filter for the metric you are plotting. Most users will not guess to do this.

(2) We may consider automatically filtering by exists, at the UI level (or perhaps at the datastore level?)

Leaving this as "discuss". cc @simianhacker @exekias @makwarth @roncohen

Examples

Kibana Visualize

Kibana Visualize (without exists - default)

Kibana Visualize (with exists)

Kibana TSVB

Kibana TSVB (without exists - default)

Note: Behavior in TSVB by default is worse than Kibana Visualize, because default ordering is by Doc count, not Metrics value. So you get an empty chart.

Other clients

Note: Other UI clients (like Grafana) will also have the same problem. Users there would have to manually apply the exists filter. Unless we figure out a way to solve this at the datastore level.