[WIP] Data Configurations of Observability Visualizations

[mengweieric]

Overview

Observability visualization nowadays provides users with only limited capabilities to visualize their data, where they have to manually write exact query every single time for rendering a visualization. That usually requires not only solid understanding of the language itself but also how different types of visualizations are visualized through what aggregation queries. Users without enough knowledges and backgrounds often feel lost in Observability visualizations as there is a gap between viewing a visualization that the they want and writing the correct query for a specific type of visualizations.

In order to address the problems stated above and provide effortless visualizing experience, Observability introduces the new data configuration UI for its visualizations.

Data Configurations

Dimensions

Dimensions section defines how the data is grouped by specified field(s), it corresponds to the group-by clause in PPL. User starts with clicking the add button to add new dimensions. While adding a new dimension, a field selector is presented to the user with a list of available fields for an index pattern. Fields are categorized as timestamp and non-timestamp types in dimension UI with different options to configure.

Timestamp
User can only select timestamp once and it has to be the first dimension. Once a timestamp is selected, no other timestamp is available for user to choose.

• timestamp selector (selector)
• intervals
  • unit selector (selector)
  • number field (number field)

Non-timestamp

• field selector (selector)

Notice the order of dimensions matters here since all these dimensions will be added after the keyword by following the order in UI. It also allows the user to drag and drop to reorder the order of dimension fields in query.

Series

Series section define the aggregation(s) over specified field(s), and user is allowed to add unlimited number of metric sections. Each metric section added should contain below items.

• field (selector)
• aggregation (selector)
• label (alias) → rename (text field)
• custom expression → override (text field)

Breakdowns

Breakdowns section defines how you split your data points by series. Some visualization may not have breakdown sections due to the nature of its data visualizations.

Query builder

Query builder fills in the gap between data configurations and aggregation query, where by specifying dimensions, series, breakdowns, and other available config options, it generates a query contains the information the user configured through the UI. #123

Visualizations which share the same data configurations

• Bar
• Timeseries
• Pie
• HeatMap
• Gauge
• Stat
• Histogram
• TreeMap
• Scatter

Visualizations which do not share the same data configurations

• Drilldown list
• Log view
• Table view
• Node graph
• Sankey Diagrams

Bar

Bar chart uses common data configurations for visualization configurations. The dimension(s) selected will be x-axis with breakdown(s) as y-axis. There are many scenarios when different number of dimensions/metrics/breakdowns are selected.

Without selecting breakdowns

Without specifying breakdown field(s), the x-axis of the bar chart for each data point is the combination of D1, D2, D3. All metrics that fall into the same D1, D2, D3 combination are displayed side by side for that combo bucket by default.

source=index | stats M1, M2 by D1, D2, D3

D1 has unique values A,B,C. D2 has unique values E,F, D3 has unique values G, H.

M1	M2	D1	D2	D3
31	13.24	A	E	G
42	17.11	A	E	H
15	7.81	B	F	H
40	6.57	C	E	G
7	1.22	C	F	G

The data for rendering plotly is

var trace1 = { 
    x: ['A,E,G', 'A,E,H', 'B,F,H', 'C,E,G', 'C,F,G'],
    y: [31,42,15,40,7]
};
var trace2 = { 
    x: ['A,E,G', 'A,E,H', 'B,F,H', 'C,E,G', 'C,F,G'],
    y: [13.24,17.11,7.81,6.57,1.22]
};

With selected breakdown field(s)

When user selects breakdown field(s), the selected dimension(s) combined will be the x-axis, and the selected breakdown field(s) combined will be the grouping rule for each x-axis datapoint.

source=index | stats M by D1, D2, D3, D4

D1 has unique values A,B,C. D2 has unique values E,F, D3 has unique values G, H, D4 has values Y, Z.

M1	D1	D2	D3	D4
31	A	E	G	Y
42	A	E	H	Y
15	B	F	H	Y
40	C	F	G	Y
7	C	F	G	Z

If user selects D1, D2 as 2 dimensions and D3, D4 as breakdowns, then in this case ‘D1,D2’ combined will be the x-axis for bucketing, ‘D3, D4’ combined will be the y-axis grouping rules and leads to the data for rendering plotly be

var trace1 = { 
    x: ['A,E', 'B,F', 'C,F'],
    y: [31, 15, 40]
};
var trace2 = { 
    x: ['A,E', 'B,F', 'C,F'],
    y: [42, null, 7]
};

X-axis is timestamp field

When timestamp is selected as a dimension, it is always be the first dimension as x-axis.

source=index | stats M1, M2 by span(timestamp, interval), D2, D3

D1 has unique values A,B,C. D2 has unique values E,F, D3 has unique values G, H.

M1	M2	T	D2	D3
31	13.24	1/1/2022	E	G
42	17.11	1/1/2022	E	H
15	7.81	1/2/2022	F	H
40	6.57	1/3/2022	E	G
7	1.22	1/3/2022	F	G

The data for rendering plotly bar chart would be

var trace1 = {
  x: ['1/1/2022', '1/3/2022'],
  y: [31,40],
  name: 'E,G,M1',
  type: 'bar'
};

var trace2 = {
  x: ['1/1/2022'],
  y: [42],
  name: 'E,H,M1',
  type: 'bar'
};

var trace3 = {
  x: ['1/2/2022'],
  y: [15],
  name: 'F,H,M1',
  type: 'bar'
};

var trace4 = {
  x: ['1/3/2022'],
  y: [7],
  name: 'F,G,M1',
  type: 'bar'
};

var trace5 = {
  x: ['1/1/2022', '1/3/2022'],
  y: [13.24, 6.57],
  name: 'E,G,M2',
  type: 'bar'
};

var trace6 = {
  x: ['1/1/2022'],
  y: [17.11],
  name: 'E,H,M2',
  type: 'bar'
};

var trace7 = {
  x: ['1/2/2022'],
  y: [7.81],
  name: 'F,H,M2',
  type: 'bar'
};

var trace8 = {
  x: ['1/3/2022'],
  y: [1.22],
  name: 'F,G,M2',
  type: 'bar'
};

Timeseries

Time series uses common data configurations for timeseries chart. The data configurations section for timeseries chart works the same as bar’s with only difference be the timestamp is always the first dimension and the only x-axis.

Pie

The pie chart displays reduced series, or values in a series, from one or more queries, as they relate to each other, in the form of slices of a pie. The arc length, area and central angle of a slice are all proportional to the slices value, as it relates to the sum of all values. This type of chart is best used when you want a quick comparison of a small set of values in an aesthetically pleasing form.

Use common data configurations for pie chart. There is no breakdown section for pie chart.

source=index | stats M1, M2 by D1, D2, D3

D1 has unique values A,B,C. D2 has unique values E,F, D3 has unique values G, H.

M1	M2	D1	D2	D3
31	13.24	A	E	G
42	17.11	A	E	H
15	7.81	B	F	H
40	6.57	C	E	G
7	1.22	C	F	G

The data for rendering is

var trace1 = { 
    labels: ['A,E,G', 'A,E,H', 'B,F,H', 'C,E,G', 'C,F,G'],
    values: [31,42,15,40,7]
};
var trace2 = { 
    labels: ['A,E,G', 'A,E,H', 'B,F,H', 'C,E,G', 'C,F,G'],
    values: [13.24,17.11,7.81,6.57,1.22]
};

Log view

The log view visualization shows log lines from data sources. Typically you would use this view next to a graph visualization / panel to display the log output of a related process. The content and the data of this type of visualizations would be the same ones in events data grid. The difference is that the expandable log details would be inline in this visualization rather than a flyout in events tab

Logs

Time (switch)
Show or hide the time column. This is the timestamp associated with the log line as reported from the data source.

Font size (number field)
Changes the size of the log font

Order (filter buttons)
Time asc / desc

HeatMap

Use common data configurations for heatmap, but supports only 2 dimensions as X and Y axis and 1 metric allowed to be selected. There will be no breakdowns section for heatmap chart.

source=index | stats M by D1, D2

D1 has values A,B,C. D2 has values E,F.

M1	D1	D2
31	A	E
42	A	F
15	B	F
40	C	E
7	C	F

var data = [
  {
    z: [[31, null, 40], [42, 15, 7]],
    x: ['A', 'B', 'C'],
    y: ['E', 'F'],
    type: 'heatmap'
  }
];

Gauge

Gauge uses common data configurations, with default number of gauges to show. User is allowed to select unlimited number of metrics, and dimensions. Each metric is essentially a gauge and there will be no breakdowns section for gauge.

No dimension selected, one metric is configured

When there’s no dimension selected, but getting 1 metrics configured, the data it received is essentially a gauge.

source=index | stats avg(bytes)

Avg(bytes)
42

The number of gauges are depended on number of metrics in the output of the query. But there will be a default number of gauge to show, and user is able to change it through config UI

No dimension selected, multiple metrics are configured

source=index | stats count(bytes), max(bytes), avg(bytes)

M1	M2	M3
19956	5697.92447	8937

Selecting multiple metrics and dimensions

source=index | stats M1, M2 by D1, D2, D3

D1 has unique values A,B,C. D2 has unique values E,F, D3 has unique values G, H.

M1	M2	D1	D2	D3
31	13.24	A	E	G
42	17.11	A	E	H
15	7.81	B	F	H
40	6.57	C	E	G
7	1.22	C	F	G

In this scenario, it would be 10 metrics with each title be name concatenating of D1,D2, D3 and M1 or M2.

Stats

Data configurations section for stat works the same as gauge’s. Each metric is essentially a stat and there will be no breakdowns section for Stat.

Histogram

Use common data configurations.

TreeMap

TreeMap charts visualize hierarchical data using nested rectangles. It uses common data configurations with no breakdowns section. The order of dimensions denotes the nested hierarchy, i.e. parent, child, grandchild levels.

source=index | stats M by D1, D2, D3, D4

D1 has unique values A,B. D2 has values E,F, D3 has values G, H, I D4 has values X, Y, Z.

M1	D1	D2	D3	D4
31	A	E	G	Y
42	A	E	H	Z
15	B	F	I	X

data = [{
      type: "treemap",
      labels: ["A", "B", "E", "F", "G", "Y", "Z", "I", "X"],
      parents: ["", "", "A", "B", "E", "G", "G", "F", "I"]
}]

Drill down list

In progress

Table view

In progress

Node graph

In progress

Scatter

In progress

Sankey Diagrams

In progress