feat: Implement quantile+sum for exponential histogram metrics

[pulpdrew]

Summary

This PR implements quantile and sum aggregations over OTEL Exponential Histogram metrics in the query builder.

max/min/sum aggregations are not supported, consistent with the histogram metrics implementation, because max/min/sum are optional in the OTEL Histogram data model. Similarly avg is not supported because it would rely on the unsupported sum aggregation.

Known limitation: Only timeseries display type is correct, inline with the existing histogram metric type. HDX-4858 will address this limitation.

Exponential Histogram Data Model

See the OTEL Docs: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram

Important notes:

1. Scale can differ between and within series. To correctly aggregate counts, the smallest scale must be used, and metrics with larger scales must be converted to the smaller series (courser buckets).
2. Offsets can (and often do) differ between data points in the same series, so calculating deltas from cumulative counts requires first aligning bucket indexes between the current and previous data point.

Query details

The quantile query is horrifying! It is broken into a few CTEs:

1. filtered_series - first step, apply filters so that later stages are only working with the relevant data
2. series_with_normalized_scale - Convert all data points to the minimum Scale value, so that they can be aggregated later. Finer-grained buckets are combined to reduce Scale. Offsets are accounted for so that the buckets being combined are aligned correctly.
3. normalized_deltas - This is a passthrough for metrics with delta temporality. For cumulative temporality, this converts cumulative counts in each data point to deltas, by comparing counts in each data point to the counts from the previous data point in the same series, via a window function. This is the most expensive part of the process, for cumulative metrics.
4. summed_buckets - Sum bucket counts across timeseries by time bucket + custom group by
5. selected_quantile_buckets - for each time bucket + custom group by, find the histogram bucket containing the requested quantile level. This requires combining negative, zero, and positive counts, in bucket order, then converting each bucket to a cumulative sum (sum of its own counts and counts in all lesser buckets).
6. metrics - Within the bucket, estimate the quantile value by interpolating between the bucket's upper and lower bounds. Use log-linear interpolation to match Prometheus behavior.

Screenshots or video

How to test locally

This can be tested locally by

1. Setting NEXT_PUBLIC_ENABLE_EXPONENTIAL_HISTOGRAMS=true in .env.local
2. Inserting some Exponential Histogram data into ClickHouse (and adding otel_metrics_exponential_histogram to your Metrics source configuration)
3. Building charts showing timeseries quantiles over the metric

Comparing the data to identical metrics sent to prometheus + grafana is a good sanity check, and agents can set this up relatively well.

References

• Linear Issue: Closes HDX-4829
• Related PRs:

[changeset-bot]

🦋 Changeset detected

Latest commit: f2aabf1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/common-utils	Patch
@hyperdx/api	Patch
@hyperdx/app	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Jul 22, 2026 2:57pm
hyperdx-storybook	Ready	Preview, Comment	Jul 22, 2026 2:57pm

[github-actions]

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

• Diff size: 622 production lines changed (Tier 2 max: < 250)
• Cross-layer change: touches backend (packages/api) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats

• Production files changed: 3
• Production lines changed: 622 (+ 2210 in test files, excluded from tier calculation)
• Branch: drew/exponential-histogram-quantile
• Author: pulpdrew

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[github-actions]

E2E Test Results

✅ All tests passed • 243 passed • 1 skipped • 1009s

Status	Count
✅ Passed	243
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

[greptile-apps]

Greptile Summary

This PR adds query-builder support for OTEL exponential histogram metrics. The main changes are:

• Count and quantile aggregation over exponential histogram tables.
• Scale normalization and cumulative-to-delta bucket handling.
• Integration fixtures and broad ClickHouse test coverage.
• Changesets for the API and common-utils packages.

Confidence Score: 4/5

This is close, but the count aggregation issue should be fixed before merging.

• The new count path can subtract a predecessor from a different cumulative stream.
• That can return incorrect counts for exponential histogram charts.
• The quantile path has much broader coverage, and no other blocking issue was selected.

packages/common-utils/src/core/histogram.ts

Important Files Changed

Filename	Overview
packages/common-utils/src/core/histogram.ts	Adds exponential histogram SQL translation for count and quantile, with a stream-isolation issue in the count predecessor window.
packages/common-utils/src/core/renderChartConfig.ts	Routes exponential histogram metric selects to the new translator and table mapping.
packages/api/src/fixtures.ts	Adds helpers for seeding exponential histogram metric points in integration tests.
packages/api/src/clickhouse/tests/renderChartConfig.int.test.ts	Adds ClickHouse integration tests for exponential histogram count, quantile, scale, reset, grouping, and filtering behavior.

_{Reviews (8): Last reviewed commit: "Merge branch 'main' into drew/exponentia..." | Re-trigger Greptile}

[github-actions]

Deep Review

Multi-agent review of the exponential-histogram quantile+sum implementation. Six reviewers ran against the net-new code in packages/common-utils/src/core/histogram.ts (functions translateExponentialHistogram*, lines 202-614), its wiring in renderChartConfig.ts (lines 2029-2113), and the app UI (AggFnSelect.tsx, MetricNameSelect.tsx). Findings below are re-graded against a ship-blocker bar; each cites a code path traceable from the diff. Environment note: git/grep were unavailable, so scope was reconstructed from the feature's source files rather than a raw diff.

🔴 P0/P1 -- must fix

• packages/common-utils/src/core/histogram.ts:587 -- sign(selected_bucket_position - negative_bucket_count - 1) evaluates in unsigned arithmetic (arrayFirstIndex→UInt32, length→UInt64), so any quantile landing in a negative bucket underflows to a large UInt64, making selected_bucket_side resolve to +1 and applying the positive-side exp2(...) interpolation to a negative bucket — a silently wrong sign and magnitude for negative-valued distributions.
  • Fix: cast the operands to signed before sign(), e.g. sign(toInt64(selected_bucket_position) - toInt64(negative_bucket_count) - 1).
  • _adversarial

🟡 P2 -- recommended

• packages/common-utils/src/core/histogram.ts:341 -- negative PositiveOffset/NegativeOffset (the common case for distributions of values < 1, e.g. sub-second latencies) flow into bitShiftRight (may perform a logical rather than arithmetic shift on signed offsets) and into range(offset, offset + length(...)) at lines 357/409/522, where a negative start or unsigned type promotion can error or allocate an oversized array.
  • Fix: normalize offsets with explicit signed floor division and wrap range() inputs in toInt64(...), then verify against a fixture whose values are < 1.
  • _{correctness, adversarial}
• packages/common-utils/src/core/histogram.ts:409 -- the bucket-index arrays never receive the Array(Int64) cast that the count arrays got (see the comment at line 447), so the if(..., emptyArrayInt64(), range(...)) branches and the UNION ALL delta branch can produce a Variant type that breaks sumMap in summed_buckets.
  • Fix: cast every range(...) index result to ::Array(Int64) in both the cumulative and delta branches, mirroring the counts fix.
  • _correctness
• packages/common-utils/src/core/histogram.ts:499 -- normalized_deltas references series_with_normalized_scale twice (temporality =2 and =1 branches) and ClickHouse inlines WITH CTEs, so the expensive per-row array downscaling runs over the full dataset twice before each copy is filtered by temporality.
  • Fix: collapse the temporality split into a single pass over the CTE and confirm with EXPLAIN PIPELINE that the downscaling operators appear once.
  • _performance
• packages/common-utils/src/__tests__/renderChartConfig.test.ts:769 -- no test sets metricType: MetricsDataType.ExponentialHistogram, so the entire new CTE chain and both dispatcher throw-paths (histogram.ts:214, histogram.ts:224) render in zero tests and there is no integration test validating numeric correctness of the hand-written SQL.
  • Fix: add SQL snapshot cases mirroring the histogram tests plus an integration test seeding known bucket layouts and asserting computed quantile/count values.
  • _{testing, correctness, adversarial}
• packages/common-utils/src/core/histogram.ts:276 -- valueAlias (from user-controlled _select.alias) is interpolated raw as "${valueAlias}" here and at line 610, and chSql interpolates plain strings unescaped, so an alias containing a double quote breaks out of the identifier and injects SQL into a saved/shared dashboard query.
  • Fix: bind it via the parameterized identifier path ${{ Identifier: valueAlias }} here, at line 610, and at renderChartConfig.ts:2103.
  • _{kieran-typescript}
• packages/app/src/components/AggFnSelect.tsx:43 -- the aggregation picker only filters increase for non-Sum metrics, leaving sum/avg/max/etc. selectable for exponential histograms even though translateExponentialHistogram supports only quantile and count and throws for everything else, producing a dead-end error chart.
  • Fix: restrict the option list to count and quantile when metricType is Histogram or ExponentialHistogram.
  • _{kieran-typescript}
• packages/common-utils/src/core/histogram.ts:251 -- cumulative-reset handling is now implemented a fourth time with semantics that diverge from the other paths (translateHistogramCount clamps with greatest(0, current - prev); this path returns the full current count on reset), so the same reset scenario yields different results across metric types and a future fix must be applied in every copy.
  • Fix: extract the temporality/reset decision into one shared SQL fragment, or document and cross-reference the intentional per-type differences.
  • _{maintainability}

🔵 P3 nitpicks (9)

• packages/common-utils/src/core/histogram.ts:337 -- a query mixing scales >= 63 apart makes bitShiftLeft(toInt64(1), scale_shift) overflow to 0, and positiveModulo(index, 0) then throws division-by-zero, failing the whole chart.
  • Fix: clamp scale_shift (e.g. least(scale_shift, 62)) or reject extreme scale spans.
• packages/common-utils/src/core/histogram.ts:594 -- a level outside [0,1] (e.g. entering 95 instead of 0.95) makes rank > total, arrayFirstIndex returns 0, the selected_bucket_position > 0 filter drops the row, and the chart is silently empty.
  • Fix: validate 0 <= level <= 1 at the dispatcher entry and throw a clear error.
• packages/common-utils/src/core/histogram.ts:330 -- series_with_normalized_scale emits normalized_scale AS Scale (shadowing the original series.Scale) and forward-references aliases defined later in the same SELECT (330/342/346 use 335/341/345), relying on undocumented ClickHouse alias resolution with no test guard.
  • Fix: rename the normalized outputs so both scales are never called Scale, order definitions before uses, and add a test asserting the resolved values.
• packages/common-utils/src/core/histogram.ts:591 -- fraction_within_bucket relies on ClickHouse returning 0 for the out-of-range subscript cumulative_counts[selected_bucket_position - 1] when the first bucket is selected, with no test exercising that path.
  • Fix: add a test whose quantile falls in the first ordered bucket.
• packages/common-utils/src/core/histogram.ts:252 -- AggregationTemporality codes 1/2 appear across multiple sites (72, 130, 252, 500, 533) but only line 130 documents which is delta vs cumulative.
  • Fix: define the mapping once (named constant or shared comment) so each use is self-documenting.
• packages/common-utils/src/core/histogram.ts:84 -- the exp-histogram path routes the time-bucket alias through FIXED_TIME_BUCKET_EXPR_ALIAS while the pre-existing histogram CTEs hardcode the `__hdx_time_bucket` literal, so renaming the constant would silently desync the two.
  • Fix: route the classic-histogram literals through the same constant.
• packages/app/src/components/AggFnSelect.tsx:36 -- the blanket // @ts-ignore on onChange({ aggFn: value }) suppresses all type errors on the line, not just the known narrowing gap.
  • Fix: replace with a scoped value as AggFnValues assertion or // @ts-expect-error with a note.
• packages/common-utils/src/core/histogram.ts:502 -- the inner subquery does SELECT * ... ORDER BY ... dragging the wide bucket arrays through a sort that the prev_row window then re-sorts on the same keys.
  • Fix: project only the columns the window needs before sorting, or drop the redundant ORDER BY after confirming ordering.
• packages/common-utils/src/core/histogram.ts:335 -- the uncorrelated (SELECT min(Scale) FROM filtered_series) drives an extra scan of the base table per CTE expansion (evaluated once, not per row — the per-row concern is unfounded).
  • Fix: compute normalized_scale in a single-row CTE and cross-join it, or accept it if the CTE double-eval fix above collapses the scans.

---

Reviewers (6): correctness, adversarial, performance, testing, maintainability, kieran-typescript.

Testing gaps:

• Negative-valued distributions (negative buckets) — the exact path corrupted by the selected_bucket_side underflow.
• Negative offsets / values < 1 (sub-second latencies) exercising bitShiftRight and range().
• Scale changing between consecutive data points of one series (cumulative→delta alignment across a scale change).
• Counter resets mid-series (StartTimeUnix change, decreasing counts) on the exponential path.
• Boundary quantiles level = 0 and level = 1, and level outside [0,1].
• Delta-temporality (=1) pass-through and the UNION ALL combination with cumulative-derived rows.
• groupBy across series carrying different Scale values, and multi-metric-name aggregation in summed_buckets.
• Custom timestampValueExpression — normalized_deltas propagates only TimeUnix, unlike the count/classic paths.

[pulpdrew]

packages/common-utils/src/core/histogram.ts:587 -- sign(selected_bucket_position - negative_bucket_count - 1) evaluates in unsigned arithmetic (arrayFirstIndex→UInt32, length→UInt64), so any quantile landing in a negative bucket underflows to a large UInt64, making selected_bucket_side resolve to +1 and applying the positive-side exp2(...) interpolation to a negative bucket — a silently wrong sign and magnitude for negative-valued distributions.

This is not true - ClickHouse promotes the subtraction to signed Int64.

[knudtty]

Some strategy related comments

[knudtty]

First off, are some of these CTEs referenced multiple times throughout the query? ClickHouse does this really weird thing where it will recalculate the CTE for each time you reference the query. There was a MATERIALIZED modifier recently added https://clickhouse.com/docs/sql-reference/statements/select/with#materialized-common-table-expressions, perhaps that could help perf on several of these? It was added in 26.4, so we'd need some conditional logic to add that modifier

[knudtty]

I see what you're saying about a lot of extra logic to support the different aggregation temporalities, I think it's likely worthwhile to ticket up an issue to create some prefetch query to determine the correct one to use for a series, cache it, and fire off the correct aggregation query. That would likely greatly help AggregationTemporality=1 queries.

[knudtty]

Looks like this one at least is used multiple times, the MATERIALIZED modifier would probably help here for ch-server >= 26.4

[knudtty]

Would prefetch queries help with these positive & negative offsets too?

[knudtty]

Schema related: I don't think we have any index on this column?

[knudtty]

Although it's referencing a previous CTE, might not help in this branch

[pulpdrew]

Great suggestions @knudtty - I am collecting potential optimizations in https://linear.app/clickhouse/issue/HDX-4865/investigate-potential-exponential-histogram-optimizations

I have tested omitting cumulative handling and negative bucket handling from the query, and the gains are minimal on large scales of data right now. The AggregationTemporality index could be useful if we can push it down into filtered_series. Ideally the selection should be either temporality 1 or 2, so I suspect one of those selections should always be empty, in which case materialization may not have much of an impact. But all of this can be tested more formally!

[greptile-apps]

Count Streams Can Mix

The new count query uses the same cumulative predecessor model as the quantile query, so each window predecessor must come from the same OTEL stream. This partition only separates rows by group, MetricName, attr_hash, and temporality. If two cumulative streams share those values but differ in stream identity fields outside the attribute hash, their rows can interleave and previous_count or previous_start_time can come from the other stream. That makes the count delta subtract the wrong baseline, so exponential histogram count charts can be over- or under-counted.

Knowledge Base Used: common-utils