[Lens] Add simple ratio calculation, similar to filter ratio

[wylieconlon]

Filter Ratio is a special type of metric that always creates a percentage of based on two sets of documents, for example "failures as a percent of all attempts." The reference implementation in TSVB is missing features, most notably missing support for KQL, and therefore we should make improvements.

Remaining open questions

• Is the concept of filter ratio understandable to users? Is there a better name or concept, such as "value ratio"?
• How exactly will users enter the numerator and denominator? Will we require typing with KQL, or can we use the single-filter form?

• The TSVB implementation of filter ratio is hard to debug when the filters are incorrect, how can Lens provide a better debugging experience?
• How does the filter ratio concept relate to math in general?

Implementation

All implementations of filter ratio in Lens depend on the App Arch team. Our preference is to implement this as a calculation using filtered metrics and math, making this a shortcut function. Shortcut functions let users achieve the same results multiple ways.

Prior edits

We previously attempted to implement a filter ratio in Lens, which we removed for both technical and user experience reasons. This screenshot showed our original implementation of filter ratio in Lens. It only supported "Count of documents." It hides the denominator query by default, and was generally confusing.

[cchaos]

@AlonaNadler We talked about this a bunch a while back and you know this thing always trips me up. Do you remember where we kind of settled for the UI of this? Did we want to show an actual equation (numerator / denominator)? I can't seem to find any sketches on my end.

[wylieconlon]

@AlonaNadler I like your suggestions for "Value Ratio" and "Advanced Ratio" which we discussed offline. "Advanced Ratio" looks a lot like what we've already implemented, so it's easy to implement the changes you've suggested. "Value Ratio" is harder to build, but might be a more important feature. Comments below.

"Advanced Ratio": I think your proposal is different in two ways from what we've done with Filter Ratio. We only allowed Filter Ratio on Documents, but you are proposing Document or Field. This is because there's a hidden part of the query that is running a count or other metric on that field/document selection and dividing them. Do you think users are used to building these kinds of hidden/nested queries like in Visualize? Overall I like this a lot, the only thing I would consider changing is whether it's always division- maybe we can support other math?

"Value Ratio": I think your proposal is pretty similar to the way that Filters are applied in the navigation, but has more complications because we are also aggregating.

It looks like loading a sample of documents for autocomplete is a prerequisite for building Value Ratio. On a query level this one is also more complicated than the Advanced Ratio because we can't nest them nicely- there has to be some kind of flattening. One of the ways we talked about to achieve Value Ratio is using the Terms aggregation and "display as percentage", would that work here?

[AlonaNadler]

We only allowed Filter Ratio on Documents, but you are proposing Document or Field.

I'm not sure I'm following. Advanced ratio is filter ratio from TSVB with improved UI/UX, for users who are familiar with filter ratio it is one of the most powerful capabilities in Kibana. The idea is to give an easy simple way for new or less trained users to do ratio using value ratio and allow power users use it as a nested option, Advanced ratio within the value ratio menu

Overall I like this a lot, the only thing I would consider changing is whether it's always division- maybe we can support other math?

I would like that as well, I think we need to have other math within our bucket script aggregation and allow users to free form their formula

One of the ways we talked about to achieve Value Ratio is using the Terms aggregation and "display as a percentage", would that work here?

I see the display as a percentage as valid in every calculation and chart, it is similar to what we are doing in the value preview in Discover and Data visualizer in ML

and have it as a display option (not formating)

Im not sure it always need to be term aggregation based (maybe it does(, imagine I have bar chart, x-axis: date histogram(timestamp), y-axis: count of documents on daily interval over the last 30 days, users can click on the show as percentage of total, the total being the number of documents for the entire 30 days and each bar is the percentage out of the total. Visually the chart will look the same it helps to compare and relate to specific days

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[cchaos]

I had started some early mocks for this and so far this is what I have. Essentially we can't fit an entire KQL bar in this tiny area so we will need to rely on popovers. The inputs would essentially be readOnly but still have an onClick that opens the popover so you can't edit the input directly.

What I don't know is how we want to (by default) represent this via text in the axis label.

[wylieconlon]

@cchaos Could you move that comment to #53328 instead? I think this issue is probably outdated. Also, what you're showing doesn't reuse any of the reusable components from the filter manager- are those worth using here?

[cchaos]

It seems this issue has more information in it whereas #53328 is quite empty. I'd opt for leaving this one open and closing the other.

Also, what you're showing doesn't reuse any of the reusable components from the filter manager- are those worth using here?

Depends on what those reusable components are... I don't know the code, I'm just thinking from a UI/UX perspective. We have the opportunity with Lens to rethink the UI for these aggregations. I don't think we should rely on reusable components if it doesn't match the proposed UI. Though it's possible to update those so all usages get improved.

[wylieconlon]

Yes, I see that this discussion is definitely adding context after reading through it again. To summarize the UX options we have from least knowledge of Elasticsearch to most:

a. The user uses a series of menus to enter a filter, which is what's used in the filter editor today
b. The user types a filter with KQL autocomplete, like the top search bar
c. The user types a filter without autocomplete, which is what TSVB supports

I think Alona's preference was for the first option, requiring the least Elasticsearch knowlege. Both A and B are reusable components.

[wylieconlon]

In the text of this issue, I was trying to compare the current thing that is called "filter ratio" in TSVB against what I think we want to in Lens to solve similar problems. To make this comparison more explicit, I will list out some of the things that we liked about TSVB, and things that we didn't like:

Liked about TSVB:

• The filter ratio calculation is a shortcut for a common query
• Numerator and denominator are clearly labeled
• Works with any metric, not just count
• Supports the Math aggregation which is more powerful
• Supports multiple ways of displaying as percent

Did not like about TSVB:

• Uses architecture that isn't shared with Visualize/Lens
• Does not solve other ratio problems- filters aren't the only ratio
• Does not use KQL, no autocomplete
• Does not use nice form to build filters
• Math aggregation is too complicated for simple uses
• But the Math aggregation is limited, doesn't work with split series
• Could support more ways of displaying as percent

Hopefully this will help us focus on defining the problem by breaking it down

@mbondyra @timroes

[flash1293]

This is possible by using formula - leaving the issue open to track the feature request for a quick function UI for this