Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KQL] Provide utility function to extract needed fields of a KQL query #180555

Closed
kertal opened this issue Apr 11, 2024 · 4 comments · Fixed by #183573
Closed

[KQL] Provide utility function to extract needed fields of a KQL query #180555

kertal opened this issue Apr 11, 2024 · 4 comments · Fixed by #183573
Assignees
Labels
Feature:KQL KQL impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews)

Comments

@kertal
Copy link
Member

kertal commented Apr 11, 2024

we don't know which fields are required upfront, right?

Right now we don't have any utilities to determine which fields are needed for the KQL statement. (I don't think they're needed for filters since we don't conditionally create different queries for filters.) But yeah, we could definitely make a function that takes a KQL statement and returns the fields that it matches. We should keep in mind that fields in KQL can also match multiple using keywords (e.g. foo.*: bar).

Originally posted by @lukasolson in #175081

This is a requirement to decouple DataView field / field caps requests from search source requests. We are aiming to reduce the amount of fields that's necessary when a search source is being fetched. Currently when there's no given data view, we initialize a new data view and this triggers a request for all fields. This should be optimized to just request the fields that are needed. For this purpose a utility function should be created, to return the necessary field names of a given KQL query, that can be used in a data view field request to just return those.

@kertal kertal added Feature:KQL KQL Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews) labels Apr 11, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@kertal kertal added impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort labels Apr 11, 2024
@lukasolson
Copy link
Member

I found out there is a utility that probably provides this functionality here:

export function getKueryFields(nodes: KueryNode[]): string[] {

cc @mattkime

@kertal
Copy link
Member Author

kertal commented May 8, 2024

@mattkime qq: is this issue still necessary? is the functionality of the utility @lukasolson sufficient? Thx

@kertal
Copy link
Member Author

kertal commented May 14, 2024

I took a brief look at the function that @lukasolson suggested, just copied and tried it out, yes, it should work, we might consider to import it to kbn-es-query

lukasolson added a commit that referenced this issue May 21, 2024
## Summary

Resolves #180555.

Adds a utility to kbn-es-query for getting the field names associated
with a KQL expression.

This utility already (mostly) existed in
x-pack/plugins/observability_solution/apm/common/utils/get_kuery_fields.ts
but didn't have test coverage for things like wildcards and nested
fields. This also updates the utility to be a little more robust in
checking the KQL node types.

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Matthew Kime <matt@mattki.me>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:KQL KQL impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Data, DataViews)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants