[Lens] Expose Elasticsearch accuracy warnings to the user

[flash1293]

Per #87115 (comment)

Kibana does not show warnings when sort by ascending results from elasticsearch include the unbounded error warning: "doc_count_error_upper_bound" : -1,

To give a concrete example how this could easily happen:

Given time-based indices, terms are not distributed evenly across the indices. Today for example "Tanya Roberts" is mentioned widely in news articles because she was reported as dead but then it was revealed she was alive. A very popular topic in today's news index. However the 80s Bond girl is very rarely mentioned, if at all, in older indices. So when you do a "rarest terms" aggregation across multiple shards the index for today would not return Tanya Roberts in the list of rarest values because she is seen as far from rare. However, another older index might have one mention of her and return that as being rare. So the final merged results for Tanya are 1 - entirely missing the large count from today's index.

In these cases we take special care to supply a doc_count_error_upper_bound to let client apps know how far out the doc counts can be. When we return -1 it means we could be majorly wrong - as in the Tanya Roberts blind spot. Lens and regular visualisations do not offer any warning when this is the case:

The warning in the response going unreported is this one:

The data/mapping to recreate this error is here. I use shard-routing to recreate the uneven spread of names typical of time-based indices:

DELETE test
PUT test
{
  "settings": {
    "number_of_shards": 4
  },
  "mappings": {
    "properties": {
      "name":{
        "type":"keyword"
      }
    }
  }
}
POST _bulk
{ "index" : { "_index" : "test"} }
{ "name" : "jim" }
{ "index" : { "_index" : "test"} }
{ "name" : "bob" }
{ "index" : { "_index" : "test"} }
{ "name" : "sue" }
{ "index" : { "_index" : "test"} }
{ "name" : "jane" }
{ "index" : { "_index" : "test", "routing": "earlier"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "tanya roberts" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z1" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z2" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z3" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z4" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z5" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z6" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z7" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z8" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z9" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z10" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z11" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z12" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z13" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z14" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z15" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z16" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z17" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z18" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z19" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z20" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z21" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z22" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z23" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z24" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z25" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z26" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z27" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z28" }
{ "index" : { "_index" : "test", "routing": "today"} }
{ "name" : "z29" }

GET test/_search
{
  "size":0,
  "aggs":{
    "terms":{
      "terms":{
        "field": "name",
        "order": {
          "_count": "asc"
        }
      }
    }
  }
}

When working with descending count sort orders the results are much less prone to inaccuracy and are more bounded error margins (most shards agree on Trump, Obama etc being the most popular) but ascending sort order inaccuracies can be wild.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[markharwood]

Thanks for opening this @flash1293
It's worth noting that doc_count_error_upper_bound values of zero are fully accurate.
When we know we are inaccurate we often are capable of knowing the worst case scenario of how far out the doc counts might be and will return this as a > 0 number.
A -1 value in the response is the extreme case where we can't even begin to estimate how inaccurate the results are.

Wording which is comprehensible to lay users is likely to be tricky but the low-level guidance for devs might be of some help

[wylieconlon]

Is this a warning, error, or information? I don't think we have an existing pattern to fit this into, so we'll need some thinking about how this would look and behave. Some questions:

• Where does the accuracy info belong in the Lens editor?
• What about the Lens embeddable?
• How does accuracy display in relation to a scrollable data table?

I would really like to avoid the previous patterns that are used in Visualize: semi-transparent warnings that obscure the chart, and toast messages which stack up on the screen when you load a dashboard. Can we come up with a more user-friendly way to display this?

[flash1293]

Not a fan of semi-transparent warning boxes either. We do have one pattern that's related (at least for the editor):

Not sure whether we need to show it in the embeddable as well. cc @MichaelMarcialis do you have an idea how to show warnings in a Lens panel on a dashboard for cases where we can render a chart, but the data might not be fully correct?

[markharwood]

There's a fair amount of stuff out there about visualizing uncertainty.

It's worth considering what we can offer users to resolve the inaccuracies. For example, a button to retrieve the exact numbers for the displayed terms. Typically we mix discovery (finding out the top/lowest N terms) and reporting (showing numerical values connected to these terms) in one request. Fixing the reporting is easy - just send another query with an include clause of the discovered terms and all data nodes will focus on filling in all the details for this select group. However, fixing the discovery part (correctly identifying the global top N terms from millions in the first place) can be much harder which could involve
a) Increasing shard_size or
b) Making multiple requests using terms pagination or composite aggs or
c) Reindexing using transforms API.

While we might not be able to solve the discovery inaccuracies, the reporting inaccuracies are easily fixable.

[MichaelMarcialis]

Not sure whether we need to show it in the embeddable as well. cc @MichaelMarcialis do you have an idea how to show warnings in a Lens panel on a dashboard for cases where we can render a chart, but the data might not be fully correct?

@flash1293: I also like the idea of utilizing the warning button and popover in the editor for this. As for providing a similar warning popover for Lens panels in Dashboard, a few possibilities come to mind (though I'm not 100% familiar with what is or isn't technically possible in dashboards). The most immediate thought would be to include an alert icon button (possibly with a count of warnings) that triggers this popover either the bottom left of the panel (as a sibling to the show/hide legend button) or possibly next to the panel header (if greater prominence is desired).

Let me know if this is something you need/want mocked up, or if you prefer to just make a POC yourself in code.

[ghudgins]

consider performance warning cases when designing warnings