[Meta][Lens] Data Modelling

[timroes]

This meta issue tracks improvements in the Data Modelling in Lens. This includes improvements/additions to supported aggregations, data calculations and transformation, as well as different types of data sources.

Basic expectations

• Filters aggregation ([Lens] Support filters aggregation #61330)
• Histogram/Range aggregation ([Lens] Create combined histogram/range aggregation for numbers #59424)
• Rate aggregation ([lens] Add "counter rate" for monotonically increasing numbers #46627)
• Support for Histogram fields ([Lens] Support for Histogram data type with sum, average #74581)

Support more aggregations & calculations

Also see supported aggregations.

Aggregations:

• Rare Terms [Lens] add rare terms support #59430)
• [Lens] Support significant terms #116421
• "Other" bucket in terms ([Lens] Default to "Other" bucket in terms operation #77749)
• Percentile ([Lens] Single-value percentile metric #74574)
• HDR percentiles ([Lens] Support HDR percentiles #98499)
• Percentile rank ([Lens] Percentile rank function #93938)
• Median ([Lens] Support median as metric aggregation #74889)
• First or last match ([Lens] Provide single-value functions to show the "First" or "Last" value of some field #55895)
• Filterable metrics [Lens] Filters for each number metric (part of filter ratio) #67228 Allow filtering of individual metrics #16672
• Min/max on dates ([Lens] Allow min/max on dates for visualizations that support dates as metrics #95489)
• Last value on dates [Lens] unable to add last value of date/string field #104787
• Top metrics instead of top hits for last value [Lens] Allow use of top metric for last value #102956
• TSVB Markdown, display the last value for a text field sorted by timestamp e.g. service.state #73966
• Histogram for histogram fields [Lens] Allow histograms on histogram fields #112391
• [Lens] Drop first buckets option #120128
• [Lens] Support Sampler & Diversified Sampler aggregations #120788
• [Lens] Extended stats for metrics (standard deviation, sum of squares, variance) #122480
• Add support for significant text aggregation #31614
• support adjacency matrix aggregation in Kibana #30161
• [Lens] Add IP range function #135883
• [Lens] Add date range function #135884

Calculations:

• Time series ([Lens] Support for all time series functions #74813)
  • Cumulative sum ([Lens] Add cumulative sum aggregation #61776)
  • Derivatives ([Lens] Add derivative function #61775)
  • Moving average ([Lens] Add moving average aggregation #61777)
  • Event rate [Lens] Discuss types of rates #62375
  • Percent growth ([Lens] Shortcut to formula percentage change as an option in differences  #62381)
  • Positive growth ([lens] Add "counter rate" for monotonically increasing numbers #46627)
  • Cumulative cardinality ([Lens] Support for cumulative cardinality metric function #75911)
  • Moving average options [Lens] Provide more options on moving average #104815
  • Instant delta (difference between last two values)
  • Scale metric by time ([Lens] Options to scale metrics into a "rate per second" or "rate per minute" #77811)
• Scale metric by whole time range ([Lens] Time scaling without date histogram #79656)
• Time offset comparisons [Lens] Compare against time offsets #68458
  • Improve time shift UX [Lens] Improve time shift's "custom" interval discoverability #102116
  • [Lens] Time shift previous period #104259
  • [Lens] previous bucket in formula  #115776
• Shortcut functions ([Lens] Shortcut functions that can be "exploded" to customize more #74800):
  • Ratio calculation ([Lens] Add simple ratio calculation, similar to filter ratio #42385)
  • Month over month calculation Month over month calculation in visualizations #35043
• Architectural requirements [Lens] Architectural requirements for calculations #68460
• Math using tinymath [Lens] Math using TinyMath #68969
  • MVP [Lens] Formula MVP #91087
• User experience for composing calculations/aggregations [Lens] User experience of composing aggregations and calculations #69215
• Technical changes to allow calculations [Lens] Change the internal API of operations to support calculations #76828
• [Lens] Static value: extends support for floating point number edge cases #118048

Formula extensions:

• Overall metrics [Lens] Formula: Overall metrics #94597
• Time offset [Lens] Formula: Time offset  #94599
• Conditionals [Lens] Formula: Conditionals #94603
• Time scaling [Lens] Formula: Time scaling #94605
• References [Lens] Formula: References #94608
• String / boolean values [Lens] Formula: String / boolean literals #94613
• Specify grouping dimensions for timeseries calculations [Lens] Formula: Specify grouping dimensions for time series functions #94609
• KQL autocomplete [Lens] Formula: KQL autocomplete in formula #94673
• Additional last value params [Lens] Formula: Allow changing sorting config for last value function #94788
• Define level of metric [Lens] Formula: Define level of metric #94789
• Improve handling of array values [Lens] Throw error on array values in formulas #102249
• Global filter for formula [Lens] Global filter for formula #103246
• Insert examples [Lens] Copy/insert action on custom formula doc examples #103738
• Date math [Lens] Formula: Date math #112851
• Fitting function [Lens] Formula: provide fitting function for missing/null values #115761
• [Lens] formula - clamp function min/max can be optional #125930
• [Lens] Math operation in formula nulls if any of the operations has null results #140829

Date Histogram

• Improve date histogram interval picking [Lens] Make changing time intervals easier and with less effort/clicks  #70597
• [Lens] Time series only for weekdays #2942

Improvements

• Make aggregation order change more intuitive [Lens] Update grouping editor to indicate the expected result of a grouping change #76038
• Support Range fields (Support ES range datatypes #12751)
• Support value count ([Lens] Allow "count of field" by supporting the value count aggregation #74910)
• Handle duplicate range/filter labels better ([Lens] Duplicate labels on the filters or ranges aggregation result in missing data #80225)
• Allow missing values on terms aggregation [Lens] Add advanced option to terms aggregation to show missing values #81513
• Improve formatting of array values [Lens] [Visualize] Formatting last_value/top_hits metric for array values #83637
• Handle array values better in xy/pie charts [Lens] Show notification when data for y column or 'size by' is not-numerical  #83655
• Client-side sorting of dimensions in all chart types [Lens] Allow client-side sorting of dimensions and legends at datasource level for all chart types #86184
• Filtering numbers based on the number range [Lens] Allow client-side bucket filtering of numbers by selecting the visible number range #86190
• [Lens] Table Allow for Filtering of Rows on the string Column Value(s) #122170
• Allow fixed interval in number histogram [Lens] Allowed fixed interval in a number histogram #93307
• Provide better guidance for granularity options [Lens] Provide better guidance for dimension cardinality (especially for top values) #95007
• Multi field top values [Lens] Allow multi field "top values" #95079
• Customize missing value label [Lens] Customize the missing value placeholder returned by terms aggregation #95540
• Include/exclude for terms [Lens] Enable include/exclude in terms aggreation #96585
• Decimal point percentiles [Lens] Allow decimal percentiles like 99.5 #98853
• [ ] Additional normalize by unit counter rate options [Lens] Additional normalize by unit counter rate interval options #101483
• Last value - first value shortcut function [Lens] Implement a last value - first value shortcut function #101500
• Allow spaces in time shift duration [Lens] allow synonyms and spaces in time shift intervals configuration #102405
• Absolute time shifts [Lens] Absolute time shifts #102493
• Allow time shift with terms [Lens] Allow time shift with terms #103063
• Allow filter per layer [Lens] Allow layer level filters in lens #103088
• Rollup restrictions [Lens] Add additional restrictions on rollup indices #104003
• [Lens] Filter last value fields by default #113140
• [Lens] Expose shard size for top values (terms) aggregations #117909
• Improve sorting with split by Bar in a bar charts are getting unsorted when adding a sub-bucket aggregation #17532
• [Lens] Extended bounds for number histogram #129308
• Collapsing bucket dimension [Lens] Summarize a break down by (collapse bucket column) #94619
• [Lens] Control auto interval increments #8099

Text Languages in Lens (via Unified Search)

• Support for SQL as a datasource ([Lens] Support query languages #127250)
• Type Elasticsearch query as JSON ([Lens] New datasource for Elasticsearch queries as JSON #59439)
• Expression datasource [Lens] Add expression data source #80869
• [Lens] Support EQL filtering #133513

Features

• Raw/Individual document visualizations ([Lens] Provide a datasource for individual documents #55897)
• Allow aggregations without metric [Lens] Datatable without metric column shows no results #64416
• Special handling of pre-aggregated data
• Semantic versioning data type Support for semantic version data type in index patterns, aggregations, and Lens #93248
• Generate filters from a query [Lens] Allow filters to be generated from the results of a query, like "greater than 95th percentile" #96154
• Usage of text fields Lens only shows keyword fields, not text fields #96576
• Predictive layers [Lens] ML predictive layers #104816
• Linear regression [Lens] Linear regression #104817
• Solution-specific datasources [Lens] Solution-specific datasources, such as machine learning or observability #104928
• Set minimum auto-interval interval on date_histogram #3787
• Ability to align date histograms with current time automatically #7556

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[fbaligand]

Hi @timroes,

To be fully exhaustive compared to classic "aggregation based" visualizations, Lens misses an equivalent for "Average/Min/Max/Sum Bucket" metric aggregation.
It would be great to add this feature in this meta issue for Lens.

[flash1293]

@fbaligand Does #94619 cover your use case?

[fbaligand]

Hi @flash1293,

Yes, it is the idea.
That said, with more possible bucket aggregations.

This is actual bucket aggregations list in "Average/Min/Max/Sum Bucket" metric aggregation:

And "Reduce op" could also be "Unique Count".

My aim is that Lens has - at least - all features possible in classic "aggregation-based" visualizations.

[timductive]

Closing old meta issues in favor of more granular, current meta issues