New saved object format for visualize

[timroes]

With the introduction of the new pipeline (#19813) for visualizations we need to change the saved object format for visualizations. The new format is supposed to look something like:

{
  title: 'vis title',
  description: 'vis description',
  pipeline: "pipeline string",
  state: {},
}

• pipeline: Will hold the pipeline expression that makes up the chart.
• state: Will be a serialized uiState or hopefully visState (see Split visualization's uiState into two states #21644) at that point

Not used anymore, see comment below

• filters: An array of filters, that will be applied to the chart (see also below).
• queries: An array of queries, that will apply to the chart (see also below).
• timeRange: The time range for this visualization. Not needed right now, but we think that there is a valid point where we might want to introduce a feature, where visualizations can have their own time range (not related to dashboard panels).

Linked filters/queries

We want to be more flexible in the long run about linking filters and queries from different places together. So each of the filters or queries element above, can either be a filter defined directly in this array (i.e. a filter you would have set in the editor) or can reference a filter, stored somewhere.

For the beginning storing filters currently will use the existing "linked save search" feature from the editor. If you would use a linked save search, the editor would add one entry to filters and one entry to queries, that references that saved search element, and thus we'll load that saved search element, when creating the request for the visualization.

Unlinking from a saved search will just remove that entries from the visualization saved object.

In the long run, this doesn't necessarily need to reference only linked searches anymore, but we could possibly link to filters and queries that are stored as their own saved object directly. The saved object format would support such a feature, which might become relevant later.

Editor state

We are not sure yet, if the new editor might need to store some state/config that's not present in the pipeline, but have no use-case for it yet. Since we can add fields to the saved object in a minor version (as long as it doesn't break without that field), we could just add a editorState field to the saved object if we actually figure out we need it, so I haven't added this to our draft above.

cc @elastic/kibana-visualizations
cc @elastic/kibana-discovery - the changed filters/queries way (and referencing) might be of interest

[markov00]

What about adding a semantic version to the saved object, better if it's the same of the kibana that created it. It can be easier to migrate them in the future, having some migration schema from one version to another.

[timroes]

@markov00 I think every saved object already has a version attached to it. I would really like a central system to manage those and also migration between those versions (decoupled from any major versions). As far as I understood, that's currently not planned for our current migration plans, @chrisdavies? So in case the current version field isn't enough for what we need, Marco, I think we will add our own one, but need to dig a bit deeper into where we use the existing version field.

[timroes]

After talking to @chrisdavies we figured out, that actually transforming the toplevel of the document, can be covered by the migration system we're currently building, so no need for a version there, but actually transforming stuff inside the pipeline might be a bit more tricky, and we actually need to think about this more, since also one version on a saved object level wouldn't be enough for that.

[timroes]

After some discussion we are currently not thinking about storing timeRange, filters and queries as separate fields in the saved object anymore, but rather have them as part of the pipeline. So a pipeline with a visualization with attached filters might look as follows:

kibana | filters filters='[..]' | esaggs ...

And the same for timeRange and queries. That makes it easier to render a saved visualization, since you don't need to preprocess some filters to e.g. merge outside filters (dashboard) with the saved filters before you can pass it into the pipeline context.

The only slight disadvantage is, that the editor would now need to parse the pipeline to extract which filters/queries/timeRange it should show, but we consider this still to be the more flexible and better working solution.

[stacey-gammon]

nit: I think we should name pipeline -> expression since I think we are moving that way terminology wise.

future thought

@monfera has been investigating element templates where multiple expressions are stored in a single object and has mentioned the expression being a DAG. We've also discussed multiple expressions, storing expressions for later use, variables, etc. I wonder if representing the expression as a single string still makes sense. Can we take that string and construct a DAG from it or will it require a different format, like a mapping of expressions, then a structure to represent the DAG?

Doesn't need to be figured out now, but when we get back to focusing on this structure, maybe we'll have more clarity on whether a single expression string continues to make the most sense. I think it would be nice if an Element and an Element template were treated no different in the UI. Like if an Element is just an Element template with only a single expression. Element Templates are a way for us to ship with more advanced default Elements. So then Element Templates become:

1. a default set built by core kibana developers that we ship with.
2. a set a solution developer can add to (pluggable).
3. a set a content creator can add to.

I'll move this conversation to the issue about element templates, but I think it's important to consider in the terms of Application Architecture because I also see Element Templates as a generic concept for everyone, not something that should be a Canvas only technology. It sounds like visualizations will be migrating to be a generic expression object, also an Application Architecture thing, not a solution specific object. Anyway, just something we should probably sync about sometime post 7.0 FF!

[monfera]

I think we should name pipeline -> expression

👍

Can we take that string and construct a DAG from it or will it require a different format, like a mapping of expressions, then a structure to represent the DAG?

I just assumed we'd go on being able to represent general graph expressions as strings, to allow simple serialization, user editing, improving expression editing assist over time etc. So it was a good point to raise - with this, the implication is that (sub)expression values can be named, and the values can be referred to elsewhere by using that name. By some linked object representation we could do away with naming, although it has its own benefits besides helping serializability, such as helping expressions be self-documenting.

(Also, the linked object representation can be bijectively mapped to a string representation, maybe with generated value labels).

I think it would be nice if an Element and an Element template were treated no different in the UI.

Regularity FTW!

I'll move this conversation to the issue about element templates, but I think it's important to consider in the terms of Application Architecture because I also see Element Templates as a generic concept for everyone, not something that should be a Canvas only technology.

Yes, so maybe we could completely split the question of the expression language evolution (if any) from templating, especially as work on the element templating got suspended for now (to allow distillation of what we want it to be for users and developers, and to focus on Canvas performance, stability and usability improvements).

Besides, we could do with the ability to reuse the result of expensive calculations (which the DAG gives us), for example, often a chart would use multiple aggregations on the same, possibly preprocessed (queried, network loaded, filtered, preaggregated, extended with computed values, ...) data. So I wouldn't mind if the expression shape were discussed separately from elements or anything render related, in an uncoupled ticket.

[stacey-gammon]

Was chatting with @njd5475. For phase 1 of Alerting actions, we probably will need to keep the old saved object format around to pass it the raw es query for creating alerts on. A secondary phase could be passing the alerting infrastructure the entire expression, which it could then execute on the server and decide whether the output data went over any given threshold.

Problem with secondary phase is that all the functions would have to be common or server side, and currently all our data fetching functions are client side only. We want to work on that but I feel like it may take too long and end up delaying alerting.

Anyway, just wanted to log this thought here so it's on our radar!

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[stratoula]

Thank you for contributing to this issue, however, we are closing this issue due to inactivity as part of a backlog grooming effort. If you believe this feature/bug should still be considered, please reopen with a comment.