allows AI assistant to get info about available indices and their fields

[ppisljar]

Summary

allow AI assistant to get into about available indices and their fields

get_dataset_info(index?: string)

if index is not passed, list of all indices is returned. if index is given fields for that index are returned.

example screenshot:

[stratoula]

I was not able to test it locally but changes look good to me. I think is a great enhancement for the ai assistant and is much more generic that it works with indices instead of dataviews (also will be super helpful for creating ESQL queries as they work with indices and not dataviews)

[stratoula]

I am playing with it. I have a question, here I am giving just a substring of my dataset. I give flights and my index is kibana_sample_data_flights

Can we make it smarter and instead of trying to find an exact match return the index(indices) which contain this string ?

[stratoula]

I tested your latest changes Peter and is much better experience now. It is much smarter, I really like it!

[dgieselaar]

should we not default to @timestamp in this case? Have you seen the LLM respond with an empty string?

[ppisljar]

for some reason lens function gets executed multiple times sometimes, where first time (or not the last time) is missing properties. seems to be a bug somewhere else in our rendering, but i wasn't able to find it, so i added a check here to make sure we only render the chart once timefield is set.

timefield should actually never be empty as it has default value set.

[dgieselaar]

for some reason lens function gets executed multiple times sometimes, where first time (or not the last time) is missing properties. seems to be a bug somewhere else in our rendering, but i wasn't able to find it, so i added a check here to make sure we only render the chart once timefield is set.

Not sure if I follow. Why is it a problem if it renders multiple times? (I assume it renders multiple times, but to the same element)

[ppisljar]

not on its own, but lens seems to be doing multiple requests then, which is a performance hit if nothing else. i think its safe to not render until we get all the required properties

[dgieselaar]

@ppisljar I made some changes - hopefully that's it for now. I no longer compress the fields - the LLM was struggling with decoding the fields and it started hallucinating. As a compromise, I don't send over the field types anymore (when asking the LLM to filter) - I don't think the LLM needs it and that saves a bunch of tokens as well. I feel like it works pretty good now!

[ppisljar]

@dgieselaar sounds good for now, if we figure out we need types we can always add them later and maybe compress the type string to a character or sth.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 4364312

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
observabilityAIAssistant	217	218	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
observabilityAIAssistant	201.2KB	204.5KB	+3.2KB

History

• 💛 Build #168087 was flaky d3bf6e1
• 💛 Build #168002 was flaky 050e971
• 💛 Build #167979 was flaky 2606cd2
• 💚 Build #166965 succeeded ee6a1ae
• 💛 Build #166923 was flaky 64a324a
• 💔 Build #166875 failed b198e47

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[stephmilovic]

LGTM on connector changes

[ppisljar]

💚 All backports created successfully

Status	Branch	Result
✅	8.11

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation