Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KQL] Add util for getting field names from KQL expression #183573

Merged
merged 12 commits into from
May 21, 2024

Conversation

lukasolson
Copy link
Member

@lukasolson lukasolson commented May 15, 2024

Summary

Resolves #180555.

Adds a utility to kbn-es-query for getting the field names associated with a KQL expression.

This utility already (mostly) existed in x-pack/plugins/observability_solution/apm/common/utils/get_kuery_fields.ts but didn't have test coverage for things like wildcards and nested fields. This also updates the utility to be a little more robust in checking the KQL node types.

Checklist

@lukasolson lukasolson added release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. Feature:KQL KQL Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews) labels May 15, 2024
@lukasolson lukasolson self-assigned this May 15, 2024
@lukasolson
Copy link
Member Author

/ci

@mattkime
Copy link
Contributor

/ci

@lukasolson lukasolson marked this pull request as ready for review May 17, 2024 00:11
@lukasolson lukasolson requested review from a team as code owners May 17, 2024 00:11
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@lukasolson lukasolson added the ci:project-deploy-observability Create an Observability project label May 17, 2024
@botelastic botelastic bot added Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team Team:obs-ux-management Observability Management User Experience Team labels May 17, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

@mattkime
Copy link
Contributor

/ci

Copy link
Contributor

@dominiqueclarke dominiqueclarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

obs-ux-management changes LGTM

Copy link
Contributor

@mattkime mattkime left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look great and work well, thanks!

@kertal
Copy link
Member

kertal commented May 21, 2024

@elasticmachine merge upstream

@kibana-ci
Copy link
Collaborator

kibana-ci commented May 21, 2024

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
apm 1806 1805 -1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/es-query 201 205 +4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
apm 3.3MB 3.3MB -139.0B

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id before after diff
module count - 5411 +5411
total size - 8.8MB +8.8MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
kbnUiSharedDeps-srcJs 3.1MB 3.1MB +731.0B
Unknown metric groups

API count

id before after diff
@kbn/es-query 261 265 +4

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @lukasolson

Copy link
Contributor

@kpatticha kpatticha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lukasolson lukasolson merged commit 564dec5 into elastic:main May 21, 2024
23 checks passed
@kibanamachine kibanamachine added v8.15.0 backport:skip This commit does not require backporting labels May 21, 2024
mattkime added a commit that referenced this pull request Jun 2, 2024
…183694)

## Summary

tldr; ES Query alert execution creates less field_caps traffic, date
fields being accessed in alert message via `fields.*` might not render.

--

This PR reduces the number of fields loaded via field caps to the
minimum required to run a query, rather than the full field list. It
adds a `createLazy` method to the Search Source Service which internally
loads fields via a DataViewLazy object and then adds them to a DataView
object. This is to minimize changes and ship code quickly - SearchSource
objects expose the DataView object they use and kibana apps may use
this. It will take time to migrate away from this since the DataView
object is used both internally and referenced externally. A key element
of this code is the ability to extract a field list from a query so a
limited (rather than complete) set of fields can be loaded.*

One side effect of loading fewer fields is that date fields available
via `fields.*` in the alert message may no longer work. Previously, all
fields were loaded including all date fields. Now, date fields are only
loaded if they're part of the query. This has been determined to be a
small corner case and an acceptable tradeoff.

Only the ES Query rule is using this new method of loading fields. While
further work is needed before wider adoption, this should prevent
significant data transfer savings via a reduction in field_caps usage.

Depends upon #183573

---

\* We don't need to load all fields to create a query, rather we need to
load all the fields where some attribute will change the output of a
query. Sometimes the translation from KQL to DSL is the same no matter
the field type (or any other attribute) and sometimes the translation is
dependent field type and other attributes. Generally speaking, we need
the latter.

There are additional complexities - we need to know which fields are
dates (and date nanos) when their values are displayed so their values
can be made uniform. In some circumstances we need to load a set of
fields due to source field exclusion - its not supported in ES so Kibana
submits a list of individual field names.

Finally, there are times where we solve a simpler problem rather than
the problem definition. Its easier to get a list of all fields
referenced in a KQL statement instead of only getting the subset we
need. A couple of extra fields is unlikely to result in performance
degradation.

---

Places where the field list is inspected -
```
packages/kbn-es-query/src/es_query/filter_matches_index.ts
packages/kbn-es-query/src/es_query/from_nested_filter.ts
packages/kbn-es-query/src/es_query/migrate_filter.ts
packages/kbn-es-query/src/kuery/functions/exists.ts
packages/kbn-es-query/src/kuery/functions/is.ts
packages/kbn-es-query/src/kuery/functions/utils/get_fields.ts
```

This looks like its worth closer examination since it looks at the
length of the field list -
https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/kuery/functions/is.ts#L110

Next steps -
- [x] Discuss above usage and make sure all cases are covered in this PR
- [x] Add statement to PR on lack of date formatting
- [x] Add test to verify reduction of fields requested

---------

Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
Co-authored-by: Lukas Olson <lukas@elastic.co>
Co-authored-by: Matthias Wilhelm <ankertal@gmail.com>
Co-authored-by: Tiago Costa <tiago.costa@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
apm:review backport:skip This commit does not require backporting ci:project-deploy-observability Create an Observability project Feature:KQL KQL release_note:plugin_api_changes Contains a Plugin API changes section for the breaking plugin API changes section. Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews) Team:obs-ux-infra_services Observability Infrastructure & Services User Experience Team Team:obs-ux-management Observability Management User Experience Team v8.15.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[KQL] Provide utility function to extract needed fields of a KQL query
9 participants