[Lens] Discuss implementation of hidden data in chart configuration

[wylieconlon]

This feature is one of the architectural requirements for calculations, but requires discussion. From the parent issue:

The most important requirement is the ability to have hidden data that is not being visualized. This data is used to calculate the final visualization. For example, if I have columns a and b, and I want to visualize a + b only, I need a way to hide the inputs. By supporting hidden data, we let the user build the visualization in steps. This makes Lens learnable.

To give some examples of analysis techniques that use hidden data:

• TSVB uses this technique when building aggregtions. For example, step 1 is to configure "count" and then step 2 is to configure "derivative of count"- which hides step 1.
• Basic math: Like in the quote above, if you want a + b without showing a, b, and a + b, then something needs to be hidden
• Positive growth rate: This is a good example because it highlights that we can solve the same problem in multiple ways, focused on the user experience
• Filter ratio: The numerator and denominator are calculated, but not shown to the user
• Time offsets: Hidden data is required to calculate the connection between time periods.

I think the way that TSVB has built a reference system that only shows the final result, while hiding all the intermediate results, is really nice but not quite right for Lens. So if we can't copy that system, we need to decide what the right system is. To give two specific examples of visualizations that we should be able to build in Lens:

Examples that should be possible in Lens

Percentage, specifically converting to a percent of the max value in a timeseries

There are so many kinds of percents, so I just picked one. I think we can use the same solution for "percent of grand total" or "percent of grand total within group". All of these calculations are possible in TSVB, as seen here:

The aggregations being used in this example are:

Aggregation	Visual encoding
Date histogram	X axis
Count	-
Overall max of Count	-
Pipeline function	Y axis

Hourly max

There are many similar analysis techniques, but I'm highlighting this one because it reduces dimensionality in a way that is helpful for analysis, especially when looking for problematic performance issues in systems. The aggregations used here are:

Aggregation	Visual encoding
Date histogram	-
Count	-
Overall max of Count	Metric

To further extend this example, I can add a "group by" aggregation on top of the date histogram, which looks like this:

Aggregation	Visual encoding
Terms	Multiple charts
Date histogram	-
Count	-
Overall max of Count	Metric

Potential user flow in Lens

The design of TSVB is really nice because it supports incremental addition and subtraction of extra aggregations, and at each incremental step the results are visible. Is it possible to have Lens still support the drag & drop system, but with more flexibility?

The main limitation for this is the design of the right side configuration panel in Lens, which currently doesn't have any way of letting users build these kinds of hidden queries:

The way I would imagine is best for users to go from this visualization to the one I described above is:

1. There is an additional set of configuration areas above the "metric" section
2. When the user clicks one of these areas, they are able to add more queries
3. In this case, the user wants a Date Histogram, so they add it. This will cause a Metric to appear for each date.
4. The user adds another aggregation for "overall max", and this reduces the number of Metrics to one and automatically hides the Date Histogram
5. The user adds a Terms aggregation. This aggregation defaults to being hidden, so now the overall max shows the "Max count for each category for each hour", which is not what they wanted
6. By un-hiding the Terms aggregation, we get the visualization shown above

This set of steps is assuming that the user wants to construct their visualization by "building up", but what if they already have an aggregation that is configured and they want to reduce its visible dimensions? Here's another potential flow:

1. The user has already configured a date histogram bar chart
2. They switch to the Metric visualization and a popover is shown asking for more information
3. The user clicks the "show multiple metrics" option, as opposed to the "show single metric" option- this copies the full configuration instead of dropping it
4. Now that the full configuration is copied, the user can click "hide" on specific dimensions which will remove them from the visualization, but keep them in the query.

Both of these examples are purely hypothetical, but I think they highlight why this problem is tricky and requires some design help.

@cchaos @AlonaNadler

Related issues:
#56696
#62385

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[flash1293]

I think we have a solution for this implemented today, so we don't need a discussion issue anymore, closing this. If that's mistaken, please reopen.