[Lens] Single-value percentile metric

[wylieconlon]

Lens should allow users to create a single-value percentile, such as 99th percentile. The user will choose a field, and the percentile will default to 95%. The user can change it using an EuiRange slider or by typing the percentile directly.

To combine multiple percentiles, the user will need to use a visualization that supports this- such as a line chart or data table.

The percentile metric supports additional options which are useful in some cases:

• HDR histogram: This is an advanced option which is most commonly used for duration fields. It is very important for a subset of users, so we should support it in Lens.
• Compression: This is an advanced option which could be described as "accuracy". I don't think it's as frequently used as HDR, and could be skipped.

cc @lukeelmers This is talking about adding some new features to the percentile aggconfig

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[lukeelmers]

@wylieconlon It sounds like in terms of support from the aggs service, all you'd need to proceed with this is to have the HDR Histogram & Compression params added as options for the percentile metric agg. Is that it, or am I missing anything?

[wylieconlon]

@lukeelmers yes, that is the request

[lukeelmers]

Thanks for confirming @wylieconlon; I've created #74687 to track the work for adding these to the AggConfigs, as I'm assuming you will want this issue to track the actual UI implementation in Lens.

[AlonaNadler]

Two related percentile functions exist today in visualize

• Percentile - users choose their field and by default, it shows the values for 1,5,10,50,95,99 Percentile
• Percentile rank - users choose the field and a value or number of values

Percentile user story in Lens

A user drag bytes into the preview:

• Lens adds a timestamp and does average(bytes),
• The user clicks on the y-axis, changes the average to percentile- By default, an input opens already populates with 95 representing 95%. Lens shows one series based on 95% of bytes.
• The user can:
  • override the 95% with another number (between 0-100)
  • add another percentile input (similar to the range multiple inputs) for example the users added another input and edit the input to show 5%.
• The results, the user has a time series chart with 2 series one representing 95% and the other 5% based on the bytes field
  User drag bytes into the y-axis (x-axis is empty) - Lens shows a bar chart with a single bar representing all documents and the overall average(bytes)
  The user changes to percentile
  The chart still shows one bar and the 95% percentile for bytes
  The user can:
  override the 95% with another number (between 0-100)
  add another percentile input (similar to the range) for example the users added another input and edit the input to show 5%.
  Without x-axis chart and sees the values of 95% and 5% as an unstacked single bar
  A chart with x-axis and y-axis (count) exists
  A users changes from count to percentile
  The users must choose a field to see a preview
  Lens shows an error explaining to add a field to use the percentile

Other requirements

• Users can fill numbers between 0-100
• Users can multiple percentiles (5%, 9%, 15%, etc…) each percentile will be a series
• Users can always edit the percentile number 5%->15%
• Users can delete percentile - the series deleted will be removed from the preview
• Ability to set custom label - currently visualize show 50th percentile of bytes that’s ok by default but in Lens we need also a way to change the label to show something else the users decide on
  Open questions:
• How/Should we tie it to HDR aggregation? @sorantis / @nehaduggal would love your thoughts on this based on how you think observability users might use HDR

Related issue (not for Lens default):
Though not the same often users using percentile want to create a bell curve from their data. This is currently not possible in Kibana #3905.
I suggest that for Lens default we focus on a percentile (simple) based on the use cases above and capture bell curve requests to the future

How can we do a bell curve chart #2704
https://discuss.elastic.co/t/plot-various-percentiles-in-kibana/132056
Interesting request - https://discuss.elastic.co/t/kibana-filtering-percentile-outliers/100507
Was requested by several customers in the past discuss https://hdrhistogram.github.io/HdrHistogram/plotFiles.html

@MichaelMarcialis @flash1293

[wylieconlon]

@AlonaNadler I think your user story there fits well with the idea I wrote above, that this is a "single-value percentile" metric. Was it intentional that you didn't include the Percentile rank in the user story?

Agreed that there is an open question about HDR percentiles.

[AlonaNadler]

@AlonaNadler I think your user story there fits well with the idea I wrote above, that this is a "single-value percentile" metric. Was it intentional that you didn't include the Percentile rank in the user story?

Yes, the main reasons I see less usage in Percentile rank and I didn't want to add complexity since it's a stretch goal for Lens by default.

[AlonaNadler]

Few suggestions to reduce the scope of this feature in the first phase:

• Lens can perform percentile in xy charts, single metric, table (not in pie and treemap)
• Users can multiple percentiles (5%, 9%, 15%, etc…) each percentile will be a series. Users will need to explicitly create multiple series one for each percentile (5% and 95%). It would be helpful in Lens if we have this capability for percentile and other reasons [Lens] Ability to clone a metric in the lens configurator #83973
  
• Ability to set custom label - currently visualize show 50th percentile of bytes that’s ok by default but in Lens we need also a way to change the label to show something else the users decide on

[wylieconlon]

@AlonaNadler I think what you're suggesting as a "reduced scope" is already what was proposed in the issue, although it would actually increase the scope if we try to disable this function for pie/treemap charts.

[flash1293]

Good point @wylieconlon , we do not restrict available operations at any point so far (average/median have the same problem as percentiles regarding pie charts). I see the point though, let's open a separate issue to discuss this in separation.