The slowness of one visualization on the dashboard can slow down the whole dashboard

[hyramduke]

Since Kibana dashboard generates one msearch request to ES for all Visualization on the dashboard, when one of the search request is super slow, everything else will wait for it too. Is there a plan to change the mechanism to break down the request individually? So that it provides better experience when there is some slow queries.

[stacey-gammon]

Thanks @hyramduke for filing this. I agree it is frustrating when the entire dashboard takes a long time to load because of a few visualizations. Unfortunately, how exactly to fix this isn't clear.

If we sent a request for each visualization individually, it might cause some visualizations to be rendered more quickly, but the entire dashboard as a whole to be rendered more slowly (from start to finish). Especially if a user is on a slow connection, multiple small requests will be slower than one giant request.

Some ideas we've tossed around:

• Batching up requests
  • But how to determine the "right" size since it will differ for every user based on their connection?
• Trying to intelligently split up slow visualizations from fast ones?
  • This would be hard to determine, but for instance, saved searches seem to be much slower than visualizations.
• Streaming the response from elasticsearch
  • Would be a huge effort and require implementation on the elasticsearch side, so unlikely.

@spalger -- Off hand, do you know anything in the courier that we can use to take advantage of batched responses? @kobelb recalled seeing something about it in courier. Maybe there is some existing functionality we could take advantage of, that we are just unaware of.

Otherwise, while I agree it'd be great to see something done about this, there is no clear path forward at this point.

[spalger]

do you know anything in the courier that we can use to take advantage of batched responses

Not sure what you mean by this

[kobelb]

@spalger I sent @stacey-gammon down this rabbit-hole because I remember seeing searchSource.onBeginSegmentedFetch and not recalling what it did. I didn't know whether we were getting partial responses from Elasticsearch and whether this callback was invoked with them. I'm likely making up functionality in courier/Elasticsearch.

[spalger]

Ah, yeah, we do multiple es request to fetch the discover data "one index at a time" and then merge them. Segmented fetch is a pretty specialized solution to a problem that doesn't really exist anymore though and I would expect it to go away in the near future.

[spalger]

It's all within the context of a single request though, it's not about fetching multiple requests in batches.

[alexfrancoeur]

@stacey-gammon @kobelb @nreese I was thinking of this from a different angle when I spoke with a customer about grouped panels earlier this week. Older versions of Kibana had this concept (#1547) and Grafana has this capability in the form of rows. I'm not necessarily advocating that we match this functionality completely, but I can see the value in manually enabling or disabling a visualization on a dashboard if the end user knows the query might be a hefty one. Grouping just makes this easier to enable / disable multiple visualizations on-demand. This puts ownership back on the dashboard creator and wouldn't require Kibana to arbitrarily load a number of visualizations or intelligently split them.

[stacey-gammon]

@alexfrancoeur - I filed #12313 which is one way we could tackle the grouping of panels and minimizing. @snide is going to pull some mocks together for us.

[stacey-gammon]

Related: #14750

[agirbal]

Any more thoughts around fixing this? Hopefully Canvas won't have the issue.
Not only it is really slowing down dashboard loads, but also makes it harder to understand which part of the dashboard is expensive. If users could see the panels loads as they become available it'd be quite intuitive what the problems may be, even if one can peek into the statistics.
Not sure the "multiple request cost" is really valid at this time, the overhead should be super minimal (most folks have a dozen panels at most, way less requests than a typical web page) and all browsers support HTTP/2 which will not present any head-of-line blocking issue. Even if you have thousands of users of a dashboard the multiple request overhead should be negligible. One valid reason could be if Kibana would want to prioritize certain panel processing but that'd make it even more opaque...
Thanks!

[agirbal]

Looks like #14750 has more info about H2 support, which seems non trivial. I got into interesting case whereby my dashboard had exactly 6 TSVB panels that were taking up all H1 connections before the msearch could kick in, delaying overall load by a lot. I'll stick to 5 TSVBs max and hope for H2 support soon, thanks!!

[lukeelmers]

Closing as _msearch was removed as the default in 7.5. There's now an advanced settings toggle courier:batchSearches to control this behavior -- more detail in the docs PR