-
Notifications
You must be signed in to change notification settings - Fork 8.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[KQL] Add util for getting field names from KQL expression #183573
Conversation
/ci |
/ci |
…to kql/get-field-names
Pinging @elastic/kibana-data-discovery (Team:DataDiscovery) |
Pinging @elastic/obs-ux-management-team (Team:obs-ux-management) |
Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services) |
/ci |
…to kql/get-field-names
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
obs-ux-management changes LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look great and work well, thanks!
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]Module Count
Public APIs missing comments
Async chunks
Canvas Sharable Runtime
Page load bundle
History
To update your PR or re-run it, just comment with: cc @lukasolson |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…183694) ## Summary tldr; ES Query alert execution creates less field_caps traffic, date fields being accessed in alert message via `fields.*` might not render. -- This PR reduces the number of fields loaded via field caps to the minimum required to run a query, rather than the full field list. It adds a `createLazy` method to the Search Source Service which internally loads fields via a DataViewLazy object and then adds them to a DataView object. This is to minimize changes and ship code quickly - SearchSource objects expose the DataView object they use and kibana apps may use this. It will take time to migrate away from this since the DataView object is used both internally and referenced externally. A key element of this code is the ability to extract a field list from a query so a limited (rather than complete) set of fields can be loaded.* One side effect of loading fewer fields is that date fields available via `fields.*` in the alert message may no longer work. Previously, all fields were loaded including all date fields. Now, date fields are only loaded if they're part of the query. This has been determined to be a small corner case and an acceptable tradeoff. Only the ES Query rule is using this new method of loading fields. While further work is needed before wider adoption, this should prevent significant data transfer savings via a reduction in field_caps usage. Depends upon #183573 --- \* We don't need to load all fields to create a query, rather we need to load all the fields where some attribute will change the output of a query. Sometimes the translation from KQL to DSL is the same no matter the field type (or any other attribute) and sometimes the translation is dependent field type and other attributes. Generally speaking, we need the latter. There are additional complexities - we need to know which fields are dates (and date nanos) when their values are displayed so their values can be made uniform. In some circumstances we need to load a set of fields due to source field exclusion - its not supported in ES so Kibana submits a list of individual field names. Finally, there are times where we solve a simpler problem rather than the problem definition. Its easier to get a list of all fields referenced in a KQL statement instead of only getting the subset we need. A couple of extra fields is unlikely to result in performance degradation. --- Places where the field list is inspected - ``` packages/kbn-es-query/src/es_query/filter_matches_index.ts packages/kbn-es-query/src/es_query/from_nested_filter.ts packages/kbn-es-query/src/es_query/migrate_filter.ts packages/kbn-es-query/src/kuery/functions/exists.ts packages/kbn-es-query/src/kuery/functions/is.ts packages/kbn-es-query/src/kuery/functions/utils/get_fields.ts ``` This looks like its worth closer examination since it looks at the length of the field list - https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/kuery/functions/is.ts#L110 Next steps - - [x] Discuss above usage and make sure all cases are covered in this PR - [x] Add statement to PR on lack of date formatting - [x] Add test to verify reduction of fields requested --------- Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co> Co-authored-by: Lukas Olson <lukas@elastic.co> Co-authored-by: Matthias Wilhelm <ankertal@gmail.com> Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Summary
Resolves #180555.
Adds a utility to kbn-es-query for getting the field names associated with a KQL expression.
This utility already (mostly) existed in x-pack/plugins/observability_solution/apm/common/utils/get_kuery_fields.ts but didn't have test coverage for things like wildcards and nested fields. This also updates the utility to be a little more robust in checking the KQL node types.
Checklist