[ML] Adds support for ES|QL in Data visualizer

[qn895]

Summary

This PR adds support for ES|QL queries in Data visualizer.

Screen.Recording.2024-01-26.at.17.21.18.mov

Changes:

• Add a new card from the Data visualizer main page

• Add a link from the ML navigation

• Added a new button to Use ES|QL

• Support for keyword, text, numeric, boolean, date, and ip fields

• Default to user's fieldFormats for fields that are dynamic generated by ES|QL, else use Data view's format

  • Default to Data view's setting (e.g. type bytes in this case for field bytes_normal_counter)
    

  • Default to user's fieldFormats formatting for dynamically generated fields (e.g. type number in this case for field avg_price)

  

• Add a new UI control to allow users to limit analysis to 5,000 - 10,000 - 100,000 - 1,000,000, rows. This speeds up fetching of the stats for big data sets and avoid potential circuit breaking exceptions.

• Break overall stats request into smaller parallel requests (which prevent time out or payload too big due by too many fields), at 10 requests at a time

• Break field stats for individual fields into more efficient batches (which prevent time out or payload too big due by too many fields), at 10 requests at at ime

• Improve error handling by propagating up the error AND the ES|QL request in both the UI and the developer's console (for better debugging)

• Improve error handling in field stats rows: If one field, or a group of fields, say 'keyword' fields fail to fetch for some reasons, it will show error for that field but not affect all other fields.

• Add deep linking in the top search bar

• More robust support for keyword fields with geo data

Todos:

• Add earliest/latest for date time fields -> Current blocker: escape special characters in esql variable names
• Fix formatting of numbers for dynamic query, where we don't know the formatting based on the data view
• Fix date time 'Update' not updating until Refresh is clicked
• Better optimization to not fetch distribution & expanded row content for pages that are not visible

Good to have:

• Investigate bringing back the +/- filter buttons (either by modifying the ES|QL query directly or by adding separate DSL filters?)

---

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Flaky Test Runner was used on any tests changed
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[qn895]

I've updated the PR to improve several things:

• Add a new UI control to allow users to limit analysis to 5,000 - 10,000 - 100,000 - 1,000,000, rows. This speeds up fetching of the stats for big data sets and avoid potential circuit breaking exceptions.
• Break overall stats request into smaller parallel requests (which prevent time out or payload too big due by too many fields), at 10 requests at a time
• Break field stats for individual fields into more efficient batches (which prevent time out or payload too big due by too many fields), at 10 requests at at ime
• Improve error handling by propagating up the error AND the ES|QL request in both the UI and the developer's console (for better debugging)
• Improve error handling in field stats rows: If one field, or a group of fields, say 'keyword' fields fail to fetch for some reasons, it will show error for that field but not affect all other fields.

• Add deep linking in the top search bar

• More robust support for Geo fields

[peteharverson]

Tested latest changes locally and on the cloud deployment. The 'limit' control is a great addition. The other optimizations are also working well - I didn't actually see any timeout warnings on the test cases I was using before.

Added one question on the limit setting here - #174188 (comment).

[peteharverson]

Tested latest changes and LGTM

[qn895]

@elasticmachine merge upstream

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1794c04
• Cloud Deployment
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-174188-1794c046f8b9

Failed CI Steps

• Jest Tests #8

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
dataVisualizer	585	603	+18
ml	1894	1895	+1
total			+19

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/ml-cancellable-search	-	8	+8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
aiops	397.8KB	397.8KB	+29.0B
dataVisualizer	617.6KB	645.3KB	+27.7KB
ml	3.6MB	3.6MB	+3.1KB
transform	402.5KB	402.5KB	+31.0B
total			+30.8KB

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
module count	-	5901	+5901
total size	-	6.5MB	+6.5MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
core	384.4KB	384.6KB	+215.0B
dataVisualizer	23.0KB	23.2KB	+148.0B
ml	78.3KB	78.4KB	+73.0B
total			+436.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/ml-cancellable-search	-	8	+8

ESLint disabled in files

id	before	after	diff
dataVisualizer	0	1	+1

ESLint disabled line counts

id	before	after	diff
dataVisualizer	51	59	+8

Total ESLint disabled count

id	before	after	diff
dataVisualizer	51	60	+9

History

• 💚 Build #189747 succeeded 27b3b10
• 💚 Build #189745 succeeded 285316c
• 💔 Build #189740 failed c613d86
• 💚 Build #189415 succeeded a8e3817

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895