Add more advanced search options to saved_objects api

[trevan]

Right now, the saved_objects api only allows you to do a simple_query_string. I have a "saved_objects" that has a boolean on it (shared or not). I'd like to make a table where the user can filter the items either by typing in text (which the normal api supports) or by specifying whether the item is shared or not. I don't believe simple_query_string can support that type of filter, especially if it is also including user-entered text.

It would be nice if there was an additional option to the saved_objects find api that lets a plugin author add additional filters.

I think this can be done by adding a new query parameter to the find api that takes an array of filters that are then added to the bool filter inside of src/server/saved_objects/client/lib/search_dsl/query_params.js.

[trevan]

@epixa, is this something that might be accepted? I'm happy to do a PR.

[epixa]

I think we can add more robust capabilities to find here, but they shouldn't be based on the elasticsearch DSL. The saved object API is intentionally an abstraction over the implementation details of how the underlying documents are stored, so exposing the ES search DSL through it is a leaky abstraction that kind of undermines the whole point of this API. We're already doing it with simple_query_string, but we'll need to change that as well, so let's let that be the last time we do it :)

A kind of crazy idea about creating some custom saved-object awareness to KQL and allowing folks to pass an entire KQL tree to find(), but that is a huge project and only a half-baked idea.

If you have an idea for a simple flag matching interface we could add to find(), then I'm all ears.

[trevan]

@epixa, what if find() could also take a kql object(?). The SavedObjectClient in the browser would convert that to the ES DSL and then send that to the saved object api on the server. The browser part of the api would be kql while the server side would be ES DSL. Or do you want the server side to also not accept ES DSL?

Another idea is that I could add an option to find() that is a key/value pair array which is translated to ES DSL on the server side. Then you could do basic flag matching and other limited term lookups.

[epixa]

The serverside is the key here, so no consumer of the saved object abstraction should deal with ES DSL.

The key/value conversion within the saved object client makes sense to me. It should be done in the client itself rather than hardcoded into the http endpoint, just like the existing searchFields stuff.

[uboness]

Are you planning on moving away from es and change the store where the dashboards are stored?

[trevan]

@epixa, ok. I'll make a PR that has a key/value array in the saved objects api.

[epixa]

@uboness No, definitely not. Elasticsearch is the only place we'll be storing kibana objects. But we want to be able to do things like the upgrade assistant without breaking all existing plugin integrations, and we're also discussing support for automatic rolling migrations on minors, which makes relying on storage level details at search time unreliable.

[uboness]

@epixa fair enough, just wanted to make sure the actual reasons for not using ES DSL are floated ("leaky abstraction" by itself is a scary argument for me... it's more about why we have an abstraction in the first place).

One note though... there definitely needs to be a solid object listing interface that supports sorting, pagination and filtering (I noticed that right now most places load all objects at once and implement pagination on the browser... not sure if that's because of a limitation we currently have in the API or not).

As for filtering... I'm not sure KQL is an appropriate abstraction... maybe it is, maybe it isn't.... it was created for the discover app and it's context is way broader than searching known kibana objects - also UX around searching/filtering objects is quite different compared to searching and exploring unstructured data. For this reason the EUI search bar implemented a much simpler narrower query lang to be used. That said, if you just support the KQL AST.... we can always implement a simpler lang (a la search bar query) that will translate to that AST.

A kind of crazy idea about creating some custom saved-object awareness to KQL and allowing folks to pass an entire KQL tree to find(), but that is a huge project and only a half-baked idea.

I wonder why is this idea "crazy"... and I also wonder why is it such a huge project? I imagine/hope the KQL implementation isn't really tied to discover and can be extracted as a higher level module in kibana, isn't it?

[epixa]

One note though... there definitely needs to be a solid object listing interface that supports sorting, pagination and filtering (I noticed that right now most places load all objects at once and implement pagination on the browser... not sure if that's because of a limitation we currently have in the API or not).

Agreed. And that should be through the saved object client's find method for sure. We don't need to expose the entirety of the elasticsearch filtering DSL through Kibana's saved object client though, which is where some sort of intermediary abstraction comes into play. There are basically infinite possibilities here, so I recommend starting with extremely limited scope. We can expand or change it later as needed.

I wonder why is this idea "crazy"...

Mostly because KQL is still an evolving idea and implementation, so creating such a systemic new coupling to it right now is kind of crazy.

To be clear, KQL is generic enough to be used in contexts outside of discover in theory, but it's still an experimental feature that they are actively thrashing on, so let's not make that any harder for them.

One particular challenge of using KQL is that it is a programmatic thing, so the featureset is large. If we accept generic KQL ASTs as inputs to the saved object client, then we need to also figure out how to represent that as custom json DSL for the saved object find REST end point, and at that point we're basically implementing elasticsearch-micro. So we'll probably still want to use the KQL AST purely as an implementation detail behind the scenes and a much more restrictive DSL on top of it.

In any case, I think the best thing we can do right now is to make incremental changes to the current find() implementation and come up with a more concrete, robust plan here when the various new abstractions in Kibana become more mature.

[trevan]

Looks like #17510 is going to accomplish the same thing.