feat: Add materialized view support (Beta)

[pulpdrew]

Closes HDX-3082

Summary

This PR back-ports support for materialized views from the EE repo. Note that this feature is in Beta, and is subject to significant changes.

This feature is intended to support:

1. Configuring AggregatingMergeTree (or SummingMergeTree) Materialized Views which are associated with a Source
2. Automatically selecting and querying an associated materialized view when a query supports it, in Chart Explorer, Custom Dashboards, the Services Dashboard, and the Search Page Histogram.
3. A UX for understanding what materialized views are available for a source, and whether (and why) it is or is not being used for a particular visualization.

Note to Reviewer(s)

This is a large PR, but the code has largely already been reviewed.

• For net-new files, types, components, and utility functions, the code does not differ from the EE repo
• Changes to the various services dashboard pages do not differ from the EE repo
• Changes to useOffsetPaginatedQuery, useChartConfig, and DBEditTimeChart differ slightly due to unrelated (to MVs) drift between this repo and the EE repo, and due to the lack of feature toggles in this repo. This is where slightly closer review would be most valuable.

Demo

Demo: MV Configuration

Config.MVs.mov

Demo: Chart Explorer

Chart.Explorer.mov

Demo: Dashboards

Dashboards.mov

Known Limitations

This feature is in Beta due to the following known limitations, which will be addressed in subsequent PRs:

1. Visualization start and end time, when not aligned with the granularity of MVs, will result in statistics based on the MV "time buckets" which fall inside the date range. This may not align exactly with the source table data which is in the selected date range.
2. Alerts do not make use of MVs, even if the associated visualization does. Due to (1), this means that alert values may not exactly match the values shown in the associated visualization.

Differences in OSS vs EE Support

• In OSS, there is a beta label on the MV configurations section
• In EE there are feature toggles to enable MV support, in OSS the feature is enabled for all teams, but will only run for sources with MVs configured.

Testing

To test, a couple of MVs can be created on the default otel_traces table, directly in ClickHouse:

Example MVs DDL

CREATE TABLE default.metrics_rollup_1m
(
    `Timestamp` DateTime,
    `ServiceName` LowCardinality(String),
    `SpanKind` LowCardinality(String),
    `StatusCode` LowCardinality(String),
    `count` SimpleAggregateFunction(sum, UInt64),
    `sum__Duration` SimpleAggregateFunction(sum, UInt64),
    `avg__Duration` AggregateFunction(avg, UInt64),
    `quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
    `min__Duration` SimpleAggregateFunction(min, UInt64),
    `max__Duration` SimpleAggregateFunction(max, UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, StatusCode, SpanKind, ServiceName);

CREATE MATERIALIZED VIEW default.metrics_rollup_1m_mv TO default.metrics_rollup_1m
(
    `Timestamp` DateTime,
    `ServiceName` LowCardinality(String),
    `SpanKind` LowCardinality(String),
    `version` LowCardinality(String),
    `StatusCode` LowCardinality(String),
    `count` UInt64,
    `sum__Duration` Int64,
    `avg__Duration` AggregateFunction(avg, UInt64),
    `quantile__Duration` AggregateFunction(quantileTDigest(0.5), UInt64),
    `min__Duration` SimpleAggregateFunction(min, UInt64),
    `max__Duration` SimpleAggregateFunction(max, UInt64)
)
AS SELECT
    toStartOfMinute(Timestamp) AS Timestamp,
    ServiceName,
    SpanKind,
    StatusCode,
    count() AS count,
    sum(Duration) AS sum__Duration,
    avgState(Duration) AS avg__Duration,
    quantileTDigestState(0.5)(Duration) AS quantile__Duration,
    minSimpleState(Duration) AS min__Duration,
    maxSimpleState(Duration) AS max__Duration
FROM default.otel_traces
GROUP BY
    Timestamp,
    ServiceName,
    SpanKind,
    StatusCode;

CREATE TABLE default.span_kind_rollup_1m
(
    `Timestamp` DateTime,
    `ServiceName` LowCardinality(String),
    `SpanKind` LowCardinality(String),
    `histogram__Duration` AggregateFunction(histogram(20), UInt64)
)
ENGINE = AggregatingMergeTree
PARTITION BY toDate(Timestamp)
ORDER BY (Timestamp, ServiceName, SpanKind);

CREATE MATERIALIZED VIEW default.span_kind_rollup_1m_mv TO default.span_kind_rollup_1m
(
    `Timestamp` DateTime,
    `ServiceName` LowCardinality(String),
    `SpanKind` LowCardinality(String),
    `histogram__Duration` AggregateFunction(histogram(20), UInt64)
)
AS SELECT
    toStartOfMinute(Timestamp) AS Timestamp,
    ServiceName,
    SpanKind,
    histogramState(20)(Duration) AS histogram__Duration
FROM default.otel_traces
GROUP BY
    Timestamp,
    ServiceName,
    SpanKind;

Then you'll need to configure the materialized views in your source settings:

Source Configuration (should auto-infer when MVs are selected)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
hyperdx-v2-oss-app	Ready	Preview, Comment	Dec 19, 2025 4:16pm

[changeset-bot]

🦋 Changeset detected

Latest commit: 13a740e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/common-utils	Minor
@hyperdx/api	Minor
@hyperdx/app	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

E2E Test Results

✅ All tests passed • 46 passed • 3 skipped • 722s

Status	Count
✅ Passed	46
❌ Failed	0
⚠️ Flaky	1
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →

[claude]

Code Review

Critical Issues

• 🔒 SQL Injection vulnerability in ServiceDashboardEndpointSidePanel.tsx:50,56 → User-controlled endpoint and service values are directly interpolated into SQL conditions without escaping. Use parameterized queries or proper escaping.

Important Issues

• ⚠️ Missing error handling in useChartConfig.tsx:276-285 → tryOptimizeConfigWithMaterializedView can throw but no try-catch wraps it. Wrap in try-catch or ensure errors propagate correctly.

• ⚠️ Race condition potential in useChartConfig.tsx:287-323 → Query cache updates during async iteration may cause stale data if query key changes mid-flight. Consider abort signal checks after each cache update.

• ⚠️ Inconsistent expression usage in ServiceDashboardEndpointSidePanel.tsx:50 → Changed from expressions.spanName to expressions.endpoint (line 50) but other files still use expressions.spanName for endpoint filtering. Verify this is intentional across the codebase.

• ⚠️ Missing validation in materializedViews.ts:173-177 → Custom count() expressions are detected but error thrown may be unclear to users. Consider more actionable error message with examples.

Notes

• ✅ Good test coverage for MV optimization logic (1603 lines of tests)
• ✅ Proper use of structuredClone to avoid reference bugs
• ✅ Beta labeling is appropriate for this feature
• ✅ Type safety maintained throughout with Zod schemas

Priority: Fix the SQL injection issue before merge.