Skip to content

Commit e08d9ba

Browse files
New Doc: Custom charts for metrics using PromQL (#260)
1 parent b38d2eb commit e08d9ba

File tree

9 files changed

+348
-15
lines changed

9 files changed

+348
-15
lines changed

docs/environment-variables.md

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ OpenObserve is configured using the following environment variables.
1515
| ZO_LOCAL_MODE | true | If local mode is set to true, OpenObserve becomes single node deployment.If it is set to false, it indicates cluster mode deployment which supports multiple nodes with different roles. For local mode one needs to configure SQLite DB, for cluster mode one needs to configure PostgreSQL (recommended) or MySQL. |
1616
| ZO_LOCAL_MODE_STORAGE | disk | Applicable only for local mode. By default, local disk is used as storage. OpenObserve supports both disk and S3 in local mode. |
1717
| ZO_NODE_ROLE | all | Node role assignment. Possible values are ingester, querier, router, compactor, alertmanager, and all. A single node can have multiple roles by specifying them as a comma-separated list. For example, compactor, alertmanager. |
18-
| ZO_NODE_ROLE_GROUP | "" | Each query-processing node can be assigned to a specific group using ZO_NODE_ROLE_GROUP. <br> - **interactive**: Handles queries triggered directly by users through the UI. <br> - **background**: Handles automated or scheduled queries, such as alerts and reports. <br> - **empty string** (default): Handles all query types. <br>
18+
| ZO_NODE_ROLE_GROUP | "" | Each query-processing node can be assigned to a specific group using ZO_NODE_ROLE_GROUP. <br>- **interactive**: Handles queries triggered directly by users through the UI. <br>- **background**: Handles automated or scheduled queries, such as alerts and reports. <br>- **empty string** (default): Handles all query types. <br>
1919
In high-load environments, alerts or reports might run large, resource-intensive queries. By assigning dedicated groups, administrators can prevent such queries from blocking or slowing down real-time user searches. |
2020
| ZO_NODE_HEARTBEAT_TTL | 30 | Time-to-live (TTL) for node heartbeats in seconds. |
2121
| ZO_INSTANCE_NAME | - | In the cluster mode, each node has a instance name. Default is instance hostname. |
@@ -87,9 +87,12 @@ In high-load environments, alerts or reports might run large, resource-intensive
8787
| ZO_MEM_TABLE_MAX_SIZE | 0 | Total size limit of all memtables. Multiple memtables exist for different organizations and stream types. Each memtable cannot exceed ZO_MAX_FILE_SIZE_IN_MEMORY, and the combined size cannot exceed this limit. If exceeded, the system returns a MemoryTableOverflowError to prevent out-of-memory conditions. Default is 50 percent of total memory. |
8888
| ZO_MEM_PERSIST_INTERVAL | 5 | Interval in seconds at which immutable memtables are persisted from memory to disk. Default is 5 seconds. |
8989
| ZO_FEATURE_SHARED_MEMTABLE_ENABLED | false | When set to true, it turns on the shared memtable feature and several organizations can use the same in-memory table instead of each organization creating its own. This helps reduce memory use when many organizations send data at the same time. It also works with older non-shared write-ahead log (WAL) files. |
90-
| ZO_MEM_TABLE_BUCKET_NUM | 1 | This setting controls how many in-memory tables OpenObserve creates, and works differently depending on whether shared memtable is enabled or disabled. <br> **When ZO_FEATURE_SHARED_MEMTABLE_ENABLED is true (shared memtable enabled)**: OpenObserve creates the specified number of shared in-memory tables that all organizations use together. <br> - **If the number is higher**: OpenObserve creates more shared tables. Each table holds data from fewer organizations. This can make data writing faster because each table handles less data. However, it also uses more memory. <br> - **If the number is lower**: OpenObserve creates fewer shared tables. Each table holds data from more organizations. This saves memory but can make data writing slightly slower when many organizations send data at the same time.
91-
<br> **When ZO_FEATURE_SHARED_MEMTABLE_ENABLED is false (shared memtable disabled)**: Each organization creates its own set of in-memory tables based on the ZO_MEM_TABLE_BUCKET_NUM value.
92-
<br> For example, if ZO_MEM_TABLE_BUCKET_NUM is set to 4, each organization will create 4 separate in-memory tables. This is particularly useful when you have only one organization, as creating multiple in-memory tables for that single organization can improve ingestion performance.|
90+
| ZO_MEM_TABLE_BUCKET_NUM | 1 | This setting controls how many in-memory tables OpenObserve creates, and works differently depending on whether shared memtable is enabled or disabled. <br> - **When ZO_FEATURE_SHARED_MEMTABLE_ENABLED is true (shared memtable enabled)**: OpenObserve creates the specified number of shared in-memory tables that all organizations use together. <br> **If the number is higher**: OpenObserve creates more shared tables. Each table holds data from fewer organizations. This can make data writing faster because each table handles less data. However, it also uses more memory. <br>
91+
**If the number is lower**: OpenObserve creates fewer shared tables. Each table holds data from more organizations. This saves memory but can make data writing slightly slower when many organizations send data at the same time. <br>- **When ZO_FEATURE_SHARED_MEMTABLE_ENABLED is false (shared memtable disabled)**:
92+
Each organization creates its own set of in-memory tables based on the ZO_MEM_TABLE_BUCKET_NUM value.
93+
<br>
94+
For example, if ZO_MEM_TABLE_BUCKET_NUM is set to 4, each organization will create 4 separate in-memory tables.
95+
This is particularly useful when you have only one organization, as creating multiple in-memory tables for that single organization can improve ingestion performance.|
9396

9497
## Indexing
9598
| Environment Variable | Default Value | Description |
433 KB
Loading

docs/images/explore-metrics.png

433 KB
Loading

docs/images/metrics-records.png

343 KB
Loading

docs/images/view-custom-chart.png

204 KB
Loading
266 KB
Loading

docs/user-guide/dashboards/custom-charts/.pages

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,4 @@ nav:
44
- Custom Charts with Flat Data: custom-charts-flat-data.md
55
- Custom Charts with Nested Data: custom-charts-nested-data.md
66
- Event Handlers and Custom Functions: custom-charts-event-handlers-and-custom-functions.md
7-
7+
- Custom charts for metrics using PromQL: custom-charts-for-metrics-using-promql.md
Lines changed: 326 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,326 @@
1+
2+
This guide explains how to build custom charts for metrics in OpenObserve using PromQL. The goal is to help new and advanced users understand how raw metrics data transforms into a fully rendered chart through predictable and repeatable steps.
3+
4+
## How metric data flows into a chart
5+
Metrics data in OpenObserve follows a fixed transformation pipeline:
6+
7+
`Metrics data in OpenObserve > PromQL query > Matrix JSON > Transform into timestamp-value pairs > Render chart`
8+
9+
This data pipeline never changes. Only two things vary:
10+
11+
- The **PromQL query**
12+
- The **JavaScript transformation logic** that prepares data based on the chart you want to build
13+
14+
The example uses the `container_cpu_time metric` and builds a time-series line chart.
15+
16+
17+
## How to build the custom chart for metrics using PromQL
18+
19+
??? "Prerequisites"
20+
### Prerequisites
21+
22+
Before building a custom chart, ensure the following:
23+
24+
- You have metrics data available in a metrics stream.
25+
- You know the basics of PromQL.
26+
- You know basic JavaScript because custom charts require writing JavaScript inside the editor.
27+
- You know the chart type you want to create and the data structure that chart expects.
28+
29+
??? "Step 1: Explore the metrics data"
30+
### Step 1: Explore the metrics data
31+
32+
OpenObserve stores metrics as time series with labels and values. To understand how your metrics look, explore them directly:
33+
34+
1. Go to **Streams**.
35+
2. Click the **Metrics** tab.
36+
3. Navigate to the metrics stream. For example, `container_cpu_time`
37+
![explore-metrics](../../../images/explore-metrics.png)
38+
4.Click **Explore.**
39+
![click-explore-metrics](../../../images/click-explore-metrics.png)
40+
This takes you to the **Logs** page and shows a time-series view:
41+
![metrics-records](../../../images/metrics-records.png)
42+
<br>
43+
44+
The two most important fields for charting are:
45+
46+
- `timestamp`
47+
- `value`
48+
49+
All charts ultimately use these two fields.
50+
51+
??? "Step 2: Decide the chart you want to build"
52+
### Step 2: Decide the chart you want to build
53+
54+
Before you write a query or JavaScript, you must decide the chart type because every chart expects a specific structure.
55+
56+
For example:
57+
58+
- A line chart requires `[timestamp, value]` pairs
59+
- A bar chart requires `[category, value]` pairs
60+
- A multi-series chart requires an array of datasets
61+
62+
Knowing the expected structure helps you prepare the right PromQL query and the right JavaScript transformation.
63+
64+
65+
??? "Step 3: Create a dashboard and select the metrics dataset"
66+
### Step 3: Create a dashboard and select the metrics dataset
67+
68+
1. In the left navigation panel, select **Dashboards** and open or create a dashboard.
69+
2. Add a panel and go to **Custom Chart** mode.
70+
3. In the **Fields** section on the left, set **Stream Type** to **metrics**.
71+
4. Select your metrics stream from the dropdown. For example: `container_cpu_time`
72+
73+
This ensures that the PromQL query will run against the correct metrics dataset.
74+
75+
??? "Step 4: Query and view your PromQL data"
76+
### Step 4: Query and view your PromQL data
77+
Before building any chart, you must query the required metric. You can view the raw PromQL response to understand the structure that your JavaScript code must transform.
78+
79+
1. Navigate to the bottom of the panel editor.
80+
2. The query editor section appears with two modes, **PromQL** and **Custom SQL**.
81+
3. Click **PromQL** to switch the editor into PromQL mode.
82+
4. In the PromQL editor, enter a PromQL expression. For example: `container_cpu_time{}`
83+
5. To understand the data structure returned by the PromQL query, paste the following JavaScript in the code editor:
84+
```js linenums="1"
85+
console.clear();
86+
console.log("=== RAW DATA ARRAY ===");
87+
console.log(data);
88+
89+
// Pretty JSON view
90+
console.log("=== RAW DATA (Pretty JSON) ===");
91+
console.log(JSON.stringify(data, null, 2));
92+
93+
// Print first query object safely
94+
if (Array.isArray(data) && data.length > 0) {
95+
console.log("=== FIRST QUERY OBJECT ===");
96+
console.dir(data[0]); // Removed depth option
97+
}
98+
99+
// Minimal valid option to avoid rendering errors
100+
option = {
101+
xAxis: { type: "time" },
102+
yAxis: { type: "value" },
103+
series: []
104+
};
105+
```
106+
6. Select the time range in the time range selector.
107+
7. Open your browser developer tools. Right-click anywhere inside the dashboard and select **Inspect**.
108+
7. Open the **Console** tab.
109+
8. In the panel editor, click **Apply**.
110+
![view-raw-metrics-data](../../../images/view-raw-metrics-data.png)
111+
You get to see the complete raw PromQL response.
112+
113+
!!! note "How to interpret it"
114+
OpenObserve returns PromQL data in the following structure:
115+
```js linenums="1"
116+
[
117+
{
118+
resultType: "matrix",
119+
result: [
120+
{
121+
metric: { ...labels... },
122+
values: [
123+
[timestamp, value],
124+
125+
...
126+
]
127+
}
128+
129+
]
130+
131+
}
132+
133+
]
134+
```
135+
Here,
136+
137+
- The outer array represents all PromQL queries in the panel. If you run one query, the array contains one item.
138+
- `resultType`: "matrix" indicates that PromQL returned time-series data.
139+
- The `result` array contains one entry for each time series in the query result.
140+
- Each metric object contains the labels that identify the series, such as `k8s_pod_name`, `container_id`, or `service_name`.
141+
- The `values` array contains the actual time-series datapoints. Each entry is `[timestamp, value]` where:
142+
143+
- `timestamp` is in Unix seconds
144+
- `value` is the metric value at that moment
145+
146+
This structure does not change. All metric visualizations in custom charts follow this same model. This is the starting point for all PromQL-based custom charts.
147+
148+
??? "Step 5: Understand how to transform the data and render the chart"
149+
### Step 5: Understand how to transform the data and render the chart
150+
Now that you have inspected the raw PromQL response, you can prepare the data and build a chart.
151+
Every PromQL-based custom chart in OpenObserve follows the same pipeline:
152+
`data > transform > series > option > chart`
153+
The following subsections explain each part in the correct order.
154+
155+
#### `data`: The raw PromQL matrix
156+
This is the starting point. `data` object is automatically available inside your custom chart editor. It holds the raw response from your PromQL query.
157+
158+
As shown in step 4, you will see the `data` object in the following structure:
159+
```js linenums="1"
160+
[
161+
{
162+
"resultType": "matrix",
163+
"result": [
164+
{
165+
"metric": {
166+
"k8s_pod_name": "o2c-openobserve-collector-agent-collector-rkggr",
167+
"container_id": "d622222c9880db586ef3a81614ef720b5030e5a4c404ff89d1616abc117cf867"
168+
},
169+
"values": [
170+
[1763035098, "39370.53"],
171+
[1763035101, "39370.53"],
172+
...
173+
]
174+
}
175+
]
176+
}
177+
]
178+
```
179+
Here:
180+
181+
- Each object inside result represents one metric series.
182+
- The metric object holds all identifying labels.
183+
- The values array holds the actual time-series data as `[timestamp, value]`.
184+
185+
186+
#### Transformation: Convert raw datapoints into chart-friendly points
187+
This is where you prepare the data for visualization. The chart that you want to build expects the data in a specific format, where each point is `[x, y]`.
188+
189+
- `x` > time (in ISO format)
190+
- `y` > numeric value
191+
192+
Perform the following conversion in JavaScript:
193+
```js linenums="1"
194+
const points = item.values.map(([timestamp, value]) => [
195+
new Date(timestamp * 1000).toISOString(),
196+
Number(value)
197+
]);
198+
```
199+
After this step, you have clean, chart-ready data such as:
200+
```js linenums="1"
201+
[
202+
["2025-11-13T09:18:00Z", 39370.53],
203+
["2025-11-13T09:18:03Z", 39370.80]
204+
]
205+
```
206+
!!! note "Note"
207+
Every chart type, whether line, bar, or scatter, starts with this transformation. Only how you display it changes later.
208+
209+
#### `series`: Build one chart series per metric
210+
`series` is an array you create in your JavaScript code. Each entry in series describes one visual line, bar set, scatter set, and so on.
211+
212+
Each entry has:
213+
214+
- A name for the legend
215+
- A type such as line
216+
- A data array with the points you want to plot
217+
218+
For example:
219+
220+
```js linenums="1"
221+
series.push({
222+
name: item.metric.k8s_pod_name || "default",
223+
type: "line",
224+
data: points,
225+
smooth: true,
226+
showSymbol: false
227+
});
228+
```
229+
230+
#### `option`: Define the final chart configuration
231+
`option` defines how the chart looks and behaves. It tells the system what axes to use, whether to display tooltips or legends, and how to organize the visual elements.
232+
```js linenums="1"
233+
option = {
234+
tooltip: { trigger: "axis" },
235+
legend: { type: "scroll" },
236+
xAxis: { type: "time", name: "Time" },
237+
yAxis: { type: "value", name: "CPU Time" },
238+
series
239+
};
240+
```
241+
The `series` array you built earlier is now linked here.
242+
243+
??? "Step 6: Transform the data and render the chart"
244+
### Step 6: Transform the data and render the chart
245+
246+
Here is the complete JavaScript code example that combines all steps mentioned in Step 5.
247+
<br>
248+
249+
**PromQL query:**
250+
```
251+
container_cpu_time{}
252+
```
253+
<br>
254+
255+
**JavaScript code:**
256+
257+
```js linenums="1"
258+
// Step 1: prepare an empty list of series
259+
const series = [];
260+
261+
// Step 2: read the PromQL response from OpenObserve
262+
if (Array.isArray(data) && data.length > 0) {
263+
const query = data[0];
264+
if (query.result && Array.isArray(query.result)) {
265+
for (const item of query.result) {
266+
if (!Array.isArray(item.values)) {
267+
continue;
268+
}
269+
270+
// Step 3: convert [timestamp, value] to [ISO time, number]
271+
const points = item.values.map(([timestamp, value]) => [
272+
new Date(timestamp * 1000).toISOString(),
273+
Number(value)
274+
]);
275+
276+
// Step 4: choose a label for the legend
277+
const name =
278+
item.metric.k8s_pod_name ||
279+
item.metric.container_id ||
280+
"unknown";
281+
282+
// Step 5: add one line series for this metric
283+
series.push({
284+
name: name,
285+
type: "line",
286+
data: points,
287+
smooth: true,
288+
showSymbol: false
289+
});
290+
291+
}
292+
293+
}
294+
295+
}
296+
297+
// Step 6: define how the chart should be drawn
298+
299+
option = {
300+
tooltip: { trigger: "axis" },
301+
legend: { type: "scroll", top: "top" },
302+
xAxis: { type: "time", name: "Time" },
303+
yAxis: { type: "value", name: "Value" },
304+
series: series
305+
};
306+
```
307+
308+
The line chart uses `[timestamp, value]` pairs and plots each metric as a line across time.
309+
310+
??? "Step 7: View the result"
311+
### Step 7: View the result
312+
![view-custom-chart](../../../images/view-custom-chart.png)
313+
Select the time range from the time range selector and click **Apply** to render your chart.
314+
315+
Each unique metric label combination will appear as a separate line.
316+
317+
!!! note "Note"
318+
You can use the same JavaScript code to create other charts that use [timestamp, value]. For example, bar charts or scatter charts. Only change the **type** in the above JavaScript code:
319+
```
320+
type: "bar"
321+
```
322+
or
323+
324+
```
325+
type: "scatter"
326+
```

docs/user-guide/logs/explain-analyze-query.md

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -116,16 +116,20 @@ The Physical Plan shows how OpenObserve executes your query, including the speci
116116
![physical-plan](../../images/physical-plan.png)
117117

118118
!!! note "Common operators you will see:"
119-
- **DataSourceExec**: Reads data from storage.
120-
- **RemoteScanExec**: Reads data from distributed partitions or remote nodes.
121-
- **FilterExec**: Applies filtering operations.
122-
- **ProjectionExec**: Handles column selection and expression computation.
123-
- **AggregateExec**: Performs aggregation operations. May show `mode=Partial` or `mode=FinalPartitioned`.
124-
- **RepartitionExec**: Redistributes data across partitions. May show `Hash([column], N)` or `RoundRobinBatch(N)`.
125-
- **CoalesceBatchesExec**: Combines data batches.
126-
- **SortExec**: Sorts data. May show `TopK(fetch=N)` for optimized sorting.
127-
- **SortPreservingMergeExec**: Merges sorted data streams.
128-
- **CooperativeExec**: Coordinates distributed execution.
119+
120+
- **DataSourceExec**: Reads data from storage
121+
- **RemoteScanExec**: Reads data from distributed partitions or remote nodes
122+
- **FilterExec**: Applies filtering operations
123+
- **ProjectionExec**: Handles column selection and expression computation
124+
- **AggregateExec**: Performs aggregation operations
125+
- May show `mode=Partial` or `mode=FinalPartitioned`
126+
- **RepartitionExec**: Redistributes data across partitions
127+
- May show `Hash([column], N)` or `RoundRobinBatch(N)`
128+
- **CoalesceBatchesExec**: Combines data batches
129+
- **SortExec**: Sorts data
130+
- May show `TopK(fetch=N)` for optimized sorting
131+
- **SortPreservingMergeExec**: Merges sorted data streams
132+
- **CooperativeExec**: Coordinates distributed execution
129133

130134
---
131135

0 commit comments

Comments
 (0)