[Meta] Common Data Access Services

[stacey-gammon]

As a foundation for a better architecture for editors, plugins and the Kibana core, we switched to rendering all visualizations in Kibana with the so far called "Canvas Expressions". This will allow us for better extensibility and building an aligned querying infrastructure for Kibana. Switching over to using those expressions for all visualizations is just the first step and there is still a lot of work ahead. A main task will be to stabilize and improve the execution of the expressions (caching, deduplicating, performance, …). We'll also planning to build a proper easy to use testing infrastructure (#22139). Since we want this expressions to be the base for all further data gathering in Kibana, building solid APIs (#25374) is a very important step to support plugin developers and solution teams.

With having the infrastructure stable, our focus will also shift to adding new functionality to the expressions (e.g. bucket script aggregations, transpose functions, etc.). We also want to work on adding new high level features to the expression execution, like terminating running execution/queries.

An important part and idea of the new infrastructure is, to create a common querying infrastructure for all components and solutions. We want to make sure, that we build expression functions that offer a proper alternative for courier (see also #20364), that we're making sure all existing visualizations (TSVB, Timelion, Graph, Canvas, etc.) are using those functions to retrieve their data. That allows us to have features like nested aggregations, parent-child relationships, etc. working aligned over all technologies. (Related issues: #14750, #18301). This also relates a lot to the future of index patterns and how those will be used across different implementations.

While SQL is rising in popularity and Canvas supports it for visualizations, we want to evaluate SQL usage in other parts of Kibana and figure out if and how SQL could improve querying and building visualizations.

Related concerns:

• Improve dashboard load performance (#14750)
• Replace courier ([Meta] Refactor Kibana querying infrastructure ("courier") #34022)
• Improve handling of slow queries/timeouts, especially in regard to frozen indices
• Allow for cancellation of queries.
• Don't reload data you don't need: Kibana improvement - don't re/load data you don't need #7215

[elasticmachine]

Pinging @elastic/kibana-app

[AlonaNadler]

Related to improving dashboard load performance and handling of frozen indices, currently, the dashboard loading time is as fast as the weakest link, which means that it only loads once all the queries return, that will be an issue with frozen indices. If even a single panel is using frozen indices, the entire dashboard will load only when the frozen index query returns, which we know will take longer. It will be good to take it under consideration when designing and try to maybe break into multiple queries instead of bulk so at least parts of the dashboard will return and users will not need to wait until frozen indices query return results.

Another issue that is raised often is how to prevent users from running bad queries in Kibana, terminating queries is one great approach, the ability to terminate the previous query once a user refresh or navigate to a different page can go a long way. We might want to think on other measurements in the data access service which can help in this goal, especially as the new GIS and the new visual builder will refresh the UI with every change unlike the traditional editor which requires to click on the play button to refresh the UI, this will result in better experience but more queries.

[aleph-zero]

We need to be clear that the problem with Kibana timeouts is not related to frozen indices. To be sure, frozen indices will exacerbate the problem, but the problem has existed for a long time. There must be some way to improve the UX w/o blocking on frozen index support.

[ppisljar]

closing this as most of the above points have been done and we have separate issues tracking any remaining work