Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[KQL] Provide utility function to extract needed fields of a KQL query #180555

Closed
kertal opened this issue Apr 11, 2024 · 4 comments · Fixed by #183573
Closed

[KQL] Provide utility function to extract needed fields of a KQL query #180555

kertal opened this issue Apr 11, 2024 · 4 comments · Fixed by #183573
Assignees
@mattkime @lukasolson
Labels
Feature:KQL KQL impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph)
Projects
Discover

Comments

@kertal
Copy link
Member

kertal commented Apr 11, 2024

we don't know which fields are required upfront, right?

Right now we don't have any utilities to determine which fields are needed for the KQL statement. (I don't think they're needed for filters since we don't conditionally create different queries for filters.) But yeah, we could definitely make a function that takes a KQL statement and returns the fields that it matches. We should keep in mind that fields in KQL can also match multiple using keywords (e.g. foo.*: bar).

Originally posted by @lukasolson in #175081

This is a requirement to decouple DataView field / field caps requests from search source requests. We are aiming to reduce the amount of fields that's necessary when a search source is being fetched. Currently when there's no given data view, we initialize a new data view and this triggers a request for all fields. This should be optimized to just request the fields that are needed. For this purpose a utility function should be created, to return the necessary field names of a given KQL query, that can be used in a data view field request to just return those.
@kertal kertal added Feature:KQL KQL Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph) labels Apr 11, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@kibanamachine kibanamachine added this to Inbox in Discover Apr 11, 2024
@kertal kertal added impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort labels Apr 11, 2024
@lukasolson
Copy link
Member

I found out there is a utility that probably provides this functionality here:

kibana/x-pack/plugins/observability_solution/apm/common/utils/get_kuery_fields.ts

Line 11 in d228f48

export function getKueryFields(nodes: KueryNode[]): string[] {

cc @mattkime

@kertal
Copy link
Member Author

kertal commented May 8, 2024

@mattkime qq: is this issue still necessary? is the functionality of the utility @lukasolson sufficient? Thx

@kertal
Copy link
Member Author

kertal commented May 14, 2024

I took a brief look at the function that @lukasolson suggested, just copied and tried it out, yes, it should work, we might consider to import it to kbn-es-query

@lukasolson lukasolson mentioned this issue May 15, 2024
Merged
2 tasks
lukasolson added a commit that referenced this issue May 21, 2024
@lukasolson @mattkime @kibanamachine
[KQL] Add util for getting field names from KQL expression (#183573)
564dec5 
## Summary

Resolves #180555.

Adds a utility to kbn-es-query for getting the field names associated
with a KQL expression.

This utility already (mostly) existed in
x-pack/plugins/observability_solution/apm/common/utils/get_kuery_fields.ts
but didn't have test coverage for things like wildcards and nested
fields. This also updates the utility to be a little more robust in
checking the KQL node types.

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

---------

Co-authored-by: Matthew Kime <matt@mattki.me>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mattkime mattkime

@lukasolson lukasolson

Labels
Feature:KQL KQL impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:medium Medium Level of Effort Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph)
Projects
No open projects
Discover
Inbox
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[KQL] Add util for getting field names from KQL expression lukasolson/kibana
4 participants
@mattkime @kertal @lukasolson @elasticmachine