Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions docs/use-cases/observability/clickstack/event_deltas.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
slug: /use-cases/observability/clickstack/event_deltas
title: 'Event Deltas with ClickStack'
sidebar_label: 'Event Deltas'
title: "Event Deltas with ClickStack"
sidebar_label: "Event Deltas"
pagination_prev: null
pagination_next: null
description: 'Event Deltas with ClickStack'
doc_type: 'guide'
description: "Event Deltas with ClickStack"
doc_type: "guide"
---

import Image from '@theme/IdealImage';
Expand All @@ -16,6 +16,7 @@ import event_deltas_selected from '@site/static/images/use-cases/observability/e
import event_deltas_issue from '@site/static/images/use-cases/observability/event_deltas_issue.png';
import event_deltas_outliers from '@site/static/images/use-cases/observability/event_deltas_outliers.png';
import event_deltas_separation from '@site/static/images/use-cases/observability/event_deltas_separation.png';
import event_deltas_customization from '@site/static/images/use-cases/observability/event_deltas_customization.png';
import event_deltas_inappropriate from '@site/static/images/use-cases/observability/event_deltas_inappropriate.png';

Event Deltas in ClickStack are a trace-focused feature that automatically analyzes the properties of traces to uncover what changed when performance regresses. By comparing the latency distributions of normal versus slow traces within a corpus, ClickStack highlights which attributes are most correlated with the difference - whether that's a new deployment version, a specific endpoint, or a particular user ID.
Expand Down Expand Up @@ -54,9 +55,31 @@ Conversely, values exclusively associated with inliers can also be interesting.

Event Deltas work by issuing two queries: one for the selected outlier area and one for the inlier area. Each query is limited to the appropriate duration and time window. A sample of events from both result sets is then inspected, and columns for which a high concentration of values appears predominantly in the outliers are identified. Columns for which 100% of a value occurs only in the outlier subset are shown first, highlighting the attributes most responsible for the observed differences.

## Customizing the graph {#customizing-the-graph}

Above the graph, you'll find controls that let you customize how the heatmap is generated. As you adjust these fields, the heatmap updates in real time, allowing you to visualize and compare relationships between any measurable value and its frequency over time.

**Default Configuration**

By default, the visualization uses:

- **Y Axis**: `Duration` — displays latency values vertically
- **Color (Z Axis)**: `count()` — represents the number of requests over time (X axis)

This setup shows latency distribution across time, with color intensity indicating how many events fall within each range.

**Adjusting Parameters**

You can modify these parameters to explore different dimensions of your data:

- **Value**: Controls what is plotted on the Y axis. For example, replace `Duration` with metrics like error rate or response size.
- **Count**: Controls the color mapping. You can switch from `count()` (number of events per bucket) to other aggregation functions such as `avg()`, `sum()`, `p95()`, or even custom expressions like `countDistinct(field)`.

<Image img={event_deltas_customization} alt="Event Deltas Customization" size="lg"/>

## Recommendations {#recommendations}

Event Deltas work best when the analysis is focused on a specific service. Latency across multiple services can vary widely, making it harder to identify the columns and values most responsible for outliers. Before enabling Event Deltas, filter spans to a set where the distribution of latencies is expected to be similar. Target analyzing sets where wide latency variation is unexpected for the most useful insights, avoiding cases where it's the norm (e.g., two different services).
Event Deltas work best when the analysis is focused on a specific service. Latency across multiple services can vary widely, making it harder to identify the columns and values most responsible for outliers. Before enabling Event Deltas, filter spans to a set where the distribution of latencies is expected to be similar. Target analyzing sets where wide latency variation is unexpected for the most useful insights, avoiding cases where it's the norm (e.g., two different services).

When selecting an area, users should aim for subsets where there is a clear distribution of slower versus faster durations, allowing the higher-latency spans to be cleanly isolated for analysis. For example, note the selected area below clearly captures a set of slower spans for analysis.

Expand Down
1 change: 1 addition & 0 deletions scripts/aspell-ignore/en/aspell-dict.txt
Original file line number Diff line number Diff line change
Expand Up @@ -489,6 +489,7 @@ HashedDictionaryThreads
HashedDictionaryThreadsActive
HashiCorp
Haversine
Heatmap
Heredoc
HexAreaKm
HexAreaM
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading