feat(llma): add clickhouse migration for tracing tables by frankh · Pull Request #70737 · PostHog/posthog

frankh · 2026-07-14T13:59:01Z

Problem

The tracing product's ClickHouse tables were created manually in prod but never captured in a migration or in local/CI schema setup. That means new environments (and local dev) don't get them, and the repo's SQL definitions had drifted from what's actually running in prod.

Changes

Adds an idempotent migration (0289_trace_spans_and_attributes) that creates the full tracing table set on the logs cluster (NodeRole.LOGS). Every statement is CREATE ... IF NOT EXISTS, so it's a no-op against the objects already present in prod and creates them fresh everywhere else.

Objects created:

Kind	Tables
Base (MergeTree)	`trace_spans`, `trace_attributes`, `trace_attributes2`, `trace_spans_kafka_metrics`
Distributed	`trace_spans_distributed`, `trace_attributes_distributed`
Materialized views → attributes	`trace_span_to_attributes`, `trace_span_to_resource_attributes`, `trace_span_to_span_attributes` (→ `trace_attributes`) and their `…2` variants (→ `trace_attributes2`)
Kafka ingestion	`kafka_trace_spans_avro` + `kafka_trace_spans_avro_mv`
Kafka metrics	`trace_spans_to_kafka_metrics_mv`

The SQL definitions in posthog/clickhouse/traces/ were rewritten to match the live prod schema exactly (dumped from system.tables), since the earlier drafts had diverged (wrong engine, extra codecs, differently-named MVs).

Note

trace_attributes is kept as the plain ReplicatedMergeTree that's live in prod today. trace_attributes2 is a new ReplicatedAggregatingMergeTree with the same shape, dual-written by the …2 MVs. Attribute counts are summed at read time so the plain engine isn't wrong, but it never collapses partial rows, which bloats storage and slows attribute queries. The cutover (backfill trace_attributes2, repoint the single trace_attributes_distributed at it, move logic.py queries onto the distributed tables, then drop the old base table and its MVs) lands in a separate PR.

Registration follows the logs-cluster convention: base and distributed tables go in schema.py for local/CI parity, while the materialized views and the Kafka table stay migration-only (creating cascade MVs on trace_spans in the shared test schema breaks the tracing tests, which hand-manage those tables). Adds the warpstream_traces named collection to local ClickHouse config so the Kafka table can be created locally.

How did you test this code?

All automated, run locally by me (Claude):

Created every object against a live local ClickHouse and confirmed each parses and builds, including trace_spans (with the _part_offset projections and is_root_span) and that trace_attributes2 reports ReplicatedAggregatingMergeTree.
posthog/clickhouse/test/test_schema.py (schema snapshots regenerated for the new base tables; the existing MV snapshots came out byte-identical after refactoring the MV bodies into shared helpers, confirming no behavior change).
The tracing product test suites (test_attribute_value_search, test_attribute_breakdown, test_flat_span_query, etc.) pass against a fresh test DB.

I did not test against prod ClickHouse directly; the migration is a pure IF NOT EXISTS no-op there by construction, validated against the prod schema dump.

Docs update

No user-facing docs change.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Frank drove this; I (Claude) wrote the migration and reconciled the repo SQL against a prod system.tables dump he provided. I invoked the /clickhouse-migrations skill for the migration conventions (node roles, IF NOT EXISTS, local-parity registration, migration-only PR scope).

Key decisions: matched prod's schema verbatim rather than the pre-existing drafts; caught that trace_attributes was plain MergeTree in prod and added the aggregating trace_attributes2 alongside it for a later cutover (single distributed table, repointed then, per Frank's call); kept the MVs and Kafka table out of schema.py after finding that cascade MVs on trace_spans broke the product's hand-managed test tables.

Created with PostHog Code

github-actions · 2026-07-14T14:04:33Z

🤖 CI report

⚠️

Playwright — 1 failed

🎭 Playwright report · View test results →

❌ 1 failed test:

Separates feature flag properties into their own tab (chromium)

These issues are not necessarily caused by your changes.
Annoyed by this section? Help fix flakies and failures and it will go green!

ℹ️ ClickHouse migration SQL — 1 migration(s)

ClickHouse migration SQL per cloud environment

unset

all

CREATE TABLE IF NOT EXISTS posthog_test.trace_spans
(
    `time_bucket` DateTime MATERIALIZED toStartOfInterval(timestamp, toIntervalHour(4)),
    `original_expiry_timestamp` DateTime64(6),
    `uuid` String,
    `team_id` Int32,
    `trace_id` String,
    `span_id` String,
    `parent_span_id` String,
    `is_root_span` Bool MATERIALIZED replaceAll(trimRight(parent_span_id, '='), 'A', '') = '',
    `trace_state` String,
    `name` LowCardinality(String),
    `kind` Int8,
    `flags` UInt32,
    `timestamp` DateTime64(6),
    `end_time` DateTime64(6),
    `observed_timestamp` DateTime64(6),
    `created_at` DateTime64(6) MATERIALIZED now(),
    `duration_nano` UInt64 MATERIALIZED toUInt64(dateDiff('microsecond', timestamp, end_time)) * 1000,
    `status_code` Int16,
    `service_name` LowCardinality(String),
    `resource_attributes` Map(LowCardinality(String), String),
    `resource_fingerprint` UInt64 MATERIALIZED cityHash64(resource_attributes),
    `instrumentation_scope` String,
    `attributes_map_str` Map(LowCardinality(String), String),
    `attributes` Map(LowCardinality(String), String) ALIAS mapApply((k, v) -> (left(k, -5), v), attributes_map_str),
    `attributes_map_float` Map(LowCardinality(String), Float64) MATERIALIZED mapFilter((k, v) -> (v IS NOT NULL), mapApply((k, v) -> (concat(left(k, -5), '__float'), toFloat64OrNull(v)), attributes_map_str)),
    `attributes_map_datetime` Map(LowCardinality(String), DateTime64(6)) MATERIALIZED mapFilter((k, v) -> (v IS NOT NULL), mapApply((k, v) -> (concat(left(k, -5), '__datetime'), parseDateTimeBestEffortOrNull(v, 6)), attributes_map_str)),
    `dropped_attributes_count` UInt32,
    `dropped_events_count` UInt32,
    `dropped_links_count` UInt32,
    `events` Array(String),
    `links` Array(String),

    -- kafka metadata
    `_partition` UInt32,
    `_topic` String,
    `_offset` UInt64,
    `_bytes_uncompressed` UInt64,
    `_bytes_compressed` UInt64,
    `_record_count` UInt64,

    INDEX idx_name name TYPE ngrambf_v1(4, 5000, 2, 0) GRANULARITY 16,
    INDEX idx_kind kind TYPE minmax GRANULARITY 4,
    INDEX idx_duration duration_nano TYPE minmax GRANULARITY 1,
    INDEX idx_status_code status_code TYPE minmax GRANULARITY 1,
    INDEX idx_timestamp_minmax timestamp TYPE minmax GRANULARITY 1,
    INDEX idx_observed_minmax observed_timestamp TYPE minmax GRANULARITY 1,
    INDEX idx_attributes_str_keys mapKeys(attributes_map_str) TYPE bloom_filter(0.01) GRANULARITY 16,
    INDEX idx_attributes_str_values mapValues(attributes_map_str) TYPE bloom_filter(0.001) GRANULARITY 16,
    INDEX idx_trace_bloom_part trace_id TYPE bloom_filter(0.00001) GRANULARITY 99999,
    INDEX idx_span_id_bloom_part span_id TYPE bloom_filter(0.00001) GRANULARITY 99999,

    -- Powers the Spans-view sparkline (spans per minute via sum(event_count)). is_root_span is a
    -- projection dimension so the Traces-view sparkline (distinct traces per minute) can serve from
    -- this projection too via sumIf(event_count, is_root_span = 1) — one root span per trace —
    -- instead of the uniqExactIf(trace_id, is_root_span = 1) raw scan it currently runs.
    PROJECTION projection_aggregate_counts
    (
        SELECT
            team_id,
            time_bucket,
            toStartOfMinute(timestamp),
            service_name,
            resource_fingerprint,
            is_root_span,
            count() AS event_count
        GROUP BY
            team_id,
            time_bucket,
            toStartOfMinute(timestamp),
            service_name,
            resource_fingerprint,
            is_root_span
    ),

    PROJECTION projection_index_span_id
    (
        SELECT _part_offset
        ORDER BY span_id
    ),

    PROJECTION projection_index_trace_id
    (
        SELECT _part_offset
        ORDER BY trace_id
    )
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/noshard/posthog.trace_spans', '{replica}-{shard}')
PARTITION BY toDate(original_expiry_timestamp)
PRIMARY KEY (team_id, time_bucket, service_name, resource_fingerprint, status_code, name, timestamp)
ORDER BY (team_id, time_bucket, service_name, resource_fingerprint, status_code, name, timestamp)
TTL original_expiry_timestamp
SETTINGS
    index_granularity_bytes = 104857600,
    index_granularity = 8192,
    ttl_only_drop_parts = 1,
    allow_part_offset_column_in_projections = 1,
    map_serialization_version = 'with_buckets'

CREATE TABLE IF NOT EXISTS posthog_test.trace_attributes
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64 DEFAULT 0,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64),
    INDEX idx_attribute_key attribute_key TYPE bloom_filter(0.01) GRANULARITY 4,
    INDEX idx_attribute_value attribute_value TYPE bloom_filter(0.01) GRANULARITY 4,
    INDEX idx_attribute_key_n3 attribute_key TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 4,
    INDEX idx_attribute_value_n3 attribute_value TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 4
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/noshard/posthog.trace_attributes', '{replica}-{shard}')
PARTITION BY toDate(original_expiry_time_bucket)
ORDER BY (team_id, attribute_type, time_bucket, resource_fingerprint, attribute_key, attribute_value)
TTL original_expiry_time_bucket
SETTINGS
    index_granularity = 8192

CREATE TABLE IF NOT EXISTS posthog_test.trace_attributes2
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64 DEFAULT 0,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64),
    INDEX idx_attribute_key attribute_key TYPE bloom_filter(0.01) GRANULARITY 4,
    INDEX idx_attribute_value attribute_value TYPE bloom_filter(0.01) GRANULARITY 4,
    INDEX idx_attribute_key_n3 attribute_key TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 4,
    INDEX idx_attribute_value_n3 attribute_value TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 4
)
ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/noshard/posthog.trace_attributes2', '{replica}-{shard}')
PARTITION BY toDate(original_expiry_time_bucket)
ORDER BY (team_id, attribute_type, time_bucket, resource_fingerprint, attribute_key, attribute_value)
TTL original_expiry_time_bucket
SETTINGS
    index_granularity = 8192

CREATE TABLE IF NOT EXISTS posthog_test.trace_spans_kafka_metrics
(
    `_partition` UInt32,
    `_topic` String,
    `max_offset` SimpleAggregateFunction(max, UInt64),
    `max_observed_timestamp` SimpleAggregateFunction(max, DateTime64(9)),
    `max_timestamp` SimpleAggregateFunction(max, DateTime64(9)),
    `max_created_at` SimpleAggregateFunction(max, DateTime64(9)),
    `max_lag` SimpleAggregateFunction(max, UInt64)
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/noshard/posthog.trace_spans_kafka_metrics', '{replica}-{shard}')
ORDER BY (_topic, _partition)
SETTINGS
    index_granularity = 8192

CREATE TABLE IF NOT EXISTS posthog_test.trace_spans_distributed AS posthog_test.trace_spans ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'trace_spans')

CREATE TABLE IF NOT EXISTS posthog_test.trace_attributes_distributed AS posthog_test.trace_attributes ENGINE = Distributed('posthog_single_shard', 'posthog_test', 'trace_attributes')

CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.trace_span_to_attributes TO posthog_test.trace_attributes
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64)
)
AS SELECT
    team_id,
    original_expiry_time_bucket,
    time_bucket,
    service_name,
    resource_fingerprint,
    attribute_key,
    attribute_value,
    'span_attribute' AS attribute_type,
    attribute_count
FROM
(
    SELECT
        team_id AS team_id,
        toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket,
        toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket,
        service_name AS service_name,
        resource_fingerprint,
        arrayJoin(mapFilter((k, v) -> ((length(k) < 256) AND (length(v) < 256)), attributes)) AS attribute,
        attribute.1 AS attribute_key,
        attribute.2 AS attribute_value,
        sumSimpleState(1) AS attribute_count
    FROM posthog_test.trace_spans
    GROUP BY
        team_id,
        original_expiry_time_bucket,
        time_bucket,
        service_name,
        resource_fingerprint,
        attribute
)

CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.trace_span_to_resource_attributes TO posthog_test.trace_attributes
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64)
)
AS SELECT
    team_id,
    original_expiry_time_bucket,
    time_bucket,
    service_name,
    resource_fingerprint,
    attribute_key,
    attribute_value,
    'span_resource_attribute' AS attribute_type,
    attribute_count
FROM
(
    SELECT
        team_id AS team_id,
        toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket,
        toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket,
        service_name AS service_name,
        resource_fingerprint,
        arrayJoin(resource_attributes) AS attribute,
        attribute.1 AS attribute_key,
        attribute.2 AS attribute_value,
        sumSimpleState(1) AS attribute_count
    FROM posthog_test.trace_spans
    GROUP BY
        team_id,
        original_expiry_time_bucket,
        time_bucket,
        service_name,
        resource_fingerprint,
        attribute
)

CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.trace_span_to_span_attributes TO posthog_test.trace_attributes
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64)
)
AS SELECT
    team_id,
    original_expiry_time_bucket,
    time_bucket,
    service_name,
    resource_fingerprint,
    attribute_key,
    attribute_value,
    'span' AS attribute_type,
    attribute_count
FROM
(
    SELECT
        team_id AS team_id,
        toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket,
        toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket,
        service_name AS service_name,
        resource_fingerprint,
        'name' AS attribute_key,
        name AS attribute_value,
        sumSimpleState(1) AS attribute_count
    FROM posthog_test.trace_spans
    GROUP BY
        team_id,
        original_expiry_time_bucket,
        time_bucket,
        service_name,
        resource_fingerprint,
        name
)

CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.trace_span_to_attributes2 TO posthog_test.trace_attributes2
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64)
)
AS SELECT
    team_id,
    original_expiry_time_bucket,
    time_bucket,
    service_name,
    resource_fingerprint,
    attribute_key,
    attribute_value,
    'span_attribute' AS attribute_type,
    attribute_count
FROM
(
    SELECT
        team_id AS team_id,
        toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket,
        toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket,
        service_name AS service_name,
        resource_fingerprint,
        arrayJoin(mapFilter((k, v) -> ((length(k) < 256) AND (length(v) < 256)), attributes)) AS attribute,
        attribute.1 AS attribute_key,
        attribute.2 AS attribute_value,
        sumSimpleState(1) AS attribute_count
    FROM posthog_test.trace_spans
    GROUP BY
        team_id,
        original_expiry_time_bucket,
        time_bucket,
        service_name,
        resource_fingerprint,
        attribute
)

CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.trace_span_to_resource_attributes2 TO posthog_test.trace_attributes2
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64)
)
AS SELECT
    team_id,
    original_expiry_time_bucket,
    time_bucket,
    service_name,
    resource_fingerprint,
    attribute_key,
    attribute_value,
    'span_resource_attribute' AS attribute_type,
    attribute_count
FROM
(
    SELECT
        team_id AS team_id,
        toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket,
        toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket,
        service_name AS service_name,
        resource_fingerprint,
        arrayJoin(resource_attributes) AS attribute,
        attribute.1 AS attribute_key,
        attribute.2 AS attribute_value,
        sumSimpleState(1) AS attribute_count
    FROM posthog_test.trace_spans
    GROUP BY
        team_id,
        original_expiry_time_bucket,
        time_bucket,
        service_name,
        resource_fingerprint,
        attribute
)

CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.trace_span_to_span_attributes2 TO posthog_test.trace_attributes2
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64)
)
AS SELECT
    team_id,
    original_expiry_time_bucket,
    time_bucket,
    service_name,
    resource_fingerprint,
    attribute_key,
    attribute_value,
    'span' AS attribute_type,
    attribute_count
FROM
(
    SELECT
        team_id AS team_id,
        toStartOfInterval(original_expiry_timestamp, toIntervalMinute(10)) AS original_expiry_time_bucket,
        toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket,
        service_name AS service_name,
        resource_fingerprint,
        'name' AS attribute_key,
        name AS attribute_value,
        sumSimpleState(1) AS attribute_count
    FROM posthog_test.trace_spans
    GROUP BY
        team_id,
        original_expiry_time_bucket,
        time_bucket,
        service_name,
        resource_fingerprint,
        name
)

CREATE TABLE IF NOT EXISTS posthog_test.kafka_trace_spans_avro
(
    `uuid` String,
    `trace_id` String,
    `span_id` String,
    `parent_span_id` String,
    `trace_state` String,
    `name` String,
    `kind` Int32,
    `flags` Int32,
    `timestamp` DateTime64(6),
    `end_time` DateTime64(6),
    `observed_timestamp` DateTime64(6),
    `service_name` String,
    `resource_attributes` Map(LowCardinality(String), String),
    `instrumentation_scope` String,
    `attributes` Map(LowCardinality(String), String),
    `dropped_attributes_count` Int32,
    `events` Array(String),
    `dropped_events_count` Int32,
    `links` Array(String),
    `dropped_links_count` Int32,
    `status_code` Int32
)
ENGINE = Kafka(warpstream_traces, kafka_topic_list = 'clickhouse_traces', kafka_group_name = 'clickhouse-traces-avro', kafka_format = 'Avro')
SETTINGS
    kafka_skip_broken_messages = 100,
    kafka_thread_per_consumer = 1,
    kafka_num_consumers = 8,
    kafka_poll_timeout_ms = 3000,
    kafka_poll_max_batch_size = 1000

CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.kafka_trace_spans_avro_mv TO posthog_test.trace_spans
(
    `uuid` String,
    `trace_id` String,
    `span_id` String,
    `parent_span_id` String,
    `trace_state` String,
    `name` String,
    `kind` Int8,
    `flags` UInt32,
    `timestamp` DateTime64(6),
    `end_time` DateTime64(6),
    `observed_timestamp` DateTime64(6),
    `service_name` String,
    `resource_attributes` Map(LowCardinality(String), String),
    `instrumentation_scope` String,
    `attributes_map_str` Map(LowCardinality(String), String),
    `dropped_attributes_count` UInt32,
    `events` Array(String),
    `dropped_events_count` UInt32,
    `links` Array(String),
    `dropped_links_count` UInt32,
    `status_code` Int16,
    `team_id` Int32,
    `original_expiry_timestamp` DateTime64(6)
)
AS SELECT
    * EXCEPT (attributes, resource_attributes, kind, flags, dropped_attributes_count, dropped_events_count, dropped_links_count, status_code),
    toInt8(kind) AS kind,
    toUInt32(flags) AS flags,
    toUInt32(dropped_attributes_count) AS dropped_attributes_count,
    toUInt32(dropped_events_count) AS dropped_events_count,
    toUInt32(dropped_links_count) AS dropped_links_count,
    toInt16(status_code) AS status_code,
    mapSort(mapApply((k, v) -> (concat(k, '__str'), JSONExtractString(v)), attributes)) AS attributes_map_str,
    mapSort(mapApply((k, v) -> (k, JSONExtractString(v)), resource_attributes)) AS resource_attributes,
    toInt32OrZero(_headers.value[indexOf(_headers.name, 'team_id')]) AS team_id,
    observed_timestamp + toIntervalDay(toInt32OrDefault(_headers.value[indexOf(_headers.name, 'retention-days')], toInt32(15))) AS original_expiry_timestamp,
    _partition,
    _topic,
    _offset,
    toInt64OrDefault(_headers.value[indexOf(_headers.name, 'record_count')], toInt64(1)) AS _record_count,
    toInt64OrDefault(_headers.value[indexOf(_headers.name, 'bytes_uncompressed')], toInt64(0)) AS _bytes_uncompressed,
    toInt64OrDefault(_headers.value[indexOf(_headers.name, 'bytes_compressed')], toInt64(0)) AS _bytes_compressed
FROM posthog_test.kafka_trace_spans_avro

CREATE MATERIALIZED VIEW IF NOT EXISTS posthog_test.trace_spans_to_kafka_metrics_mv TO posthog_test.trace_spans_kafka_metrics
(
    `_partition` UInt64,
    `_topic` LowCardinality(String),
    `max_offset` SimpleAggregateFunction(max, UInt64),
    `max_observed_timestamp` SimpleAggregateFunction(max, DateTime64(6)),
    `max_timestamp` SimpleAggregateFunction(max, DateTime64(6)),
    `max_created_at` SimpleAggregateFunction(max, DateTime),
    `max_lag` SimpleAggregateFunction(max, Decimal(18, 6))
)
AS SELECT
    _partition,
    _topic,
    maxSimpleState(_offset) AS max_offset,
    maxSimpleState(observed_timestamp) AS max_observed_timestamp,
    maxSimpleState(timestamp) AS max_timestamp,
    maxSimpleState(now()) AS max_created_at,
    maxSimpleState(now() - observed_timestamp) AS max_lag
FROM posthog_test.trace_spans
GROUP BY _partition, _topic

US, EU, DEV

logs

CREATE TABLE IF NOT EXISTS posthog_test.trace_spans
(
    `time_bucket` DateTime MATERIALIZED toStartOfInterval(timestamp, toIntervalHour(4)),
    `original_expiry_timestamp` DateTime64(6),
    `uuid` String,
    `team_id` Int32,
    `trace_id` String,
    `span_id` String,
    `parent_span_id` String,
    `is_root_span` Bool MATERIALIZED replaceAll(trimRight(parent_span_id, '='), 'A', '') = '',
    `trace_state` String,
    `name` LowCardinality(String),
    `kind` Int8,
    `flags` UInt32,
    `timestamp` DateTime64(6),
    `end_time` DateTime64(6),
    `observed_timestamp` DateTime64(6),
    `created_at` DateTime64(6) MATERIALIZED now(),
    `duration_nano` UInt64 MATERIALIZED toUInt64(dateDiff('microsecond', timestamp, end_time)) * 1000,
    `status_code` Int16,
    `service_name` LowCardinality(String),
    `resource_attributes` Map(LowCardinality(String), String),
    `resource_fingerprint` UInt64 MATERIALIZED cityHash64(resource_attributes),
    `instrumentation_scope` String,
    `attributes_map_str` Map(LowCardinality(String), String),
    `attributes` Map(LowCardinality(String), String) ALIAS mapApply((k, v) -> (left(k, -5), v), attributes_map_str),
    `attributes_map_float` Map(LowCardinality(String), Float64) MATERIALIZED mapFilter((k, v) -> (v IS NOT NULL), mapApply((k, v) -> (concat(left(k, -5), '__float'), toFloat64OrNull(v)), attributes_map_str)),
    `attributes_map_datetime` Map(LowCardinality(String), DateTime64(6)) MATERIALIZED mapFilter((k, v) -> (v IS NOT NULL), mapApply((k, v) -> (concat(left(k, -5), '__datetime'), parseDateTimeBestEffortOrNull(v, 6)), attributes_map_str)),
    `dropped_attributes_count` UInt32,
    `dropped_events_count` UInt32,
    `dropped_links_count` UInt32,
    `events` Array(String),
    `links` Array(String),

    -- kafka metadata
    `_partition` UInt32,
    `_topic` String,
    `_offset` UInt64,
    `_bytes_uncompressed` UInt64,
    `_bytes_compressed` UInt64,
    `_record_count` UInt64,

    INDEX idx_name name TYPE ngrambf_v1(4, 5000, 2, 0) GRANULARITY 16,
    INDEX idx_kind kind TYPE minmax GRANULARITY 4,
    INDEX idx_duration duration_nano TYPE minmax GRANULARITY 1,
    INDEX idx_status_code status_code TYPE minmax GRANULARITY 1,
    INDEX idx_timestamp_minmax timestamp TYPE minmax GRANULARITY 1,
    INDEX idx_observed_minmax observed_timestamp TYPE minmax GRANULARITY 1,
    INDEX idx_attributes_str_keys mapKeys(attributes_map_str) TYPE bloom_filter(0.01) GRANULARITY 16,
    INDEX idx_attributes_str_values mapValues(attributes_map_str) TYPE bloom_filter(0.001) GRANULARITY 16,
    INDEX idx_trace_bloom_part trace_id TYPE bloom_filter(0.00001) GRANULARITY 99999,
    INDEX idx_span_id_bloom_part span_id TYPE bloom_filter(0.00001) GRANULARITY 99999,

    -- Powers the Spans-view sparkline (spans per minute via sum(event_count)). is_root_span is a
    -- projection dimension so the Traces-view sparkline (distinct traces per minute) can serve from
    -- this projection too via sumIf(event_count, is_root_span = 1) — one root span per trace —
    -- instead of the uniqExactIf(trace_id, is_root_span = 1) raw scan it currently runs.
    PROJECTION projection_aggregate_counts
    (
        SELECT
            team_id,
            time_bucket,
            toStartOfMinute(timestamp),
            service_name,
            resource_fingerprint,
            is_root_span,
            count() AS event_count
        GROUP BY
            team_id,
            time_bucket,
            toStartOfMinute(timestamp),
            service_name,
            resource_fingerprint,
            is_root_span
    ),

    PROJECTION projection_index_span_id
    (
        SELECT _part_offset
        ORDER BY span_id
    ),

    PROJECTION projection_index_trace_id
    (
        SELECT _part_offset
        ORDER BY trace_id
    )
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/noshard/posthog.trace_spans', '{replica}-{shard}')
PARTITION BY toDate(original_expiry_timestamp)
PRIMARY KEY (team_id, time_bucket, service_name, resource_fingerprint, status_code, name, timestamp)
ORDER BY (team_id, time_bucket, service_name, resource_fingerprint, status_code, name, timestamp)
TTL original_expiry_timestamp
SETTINGS
    index_granularity_bytes = 104857600,
    index_granularity = 8192,
    ttl_only_drop_parts = 1,
    allow_part_offset_column_in_projections = 1,
    map_serialization_version = 'with_buckets'

CREATE TABLE IF NOT EXISTS posthog_test.trace_attributes
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64 DEFAULT 0,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64),
    INDEX idx_attribute_key attribute_key TYPE bloom_filter(0.01) GRANULARITY 4,
    INDEX idx_attribute_value attribute_value TYPE bloom_filter(0.01) GRANULARITY 4,
    INDEX idx_attribute_key_n3 attribute_key TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 4,
    INDEX idx_attribute_value_n3 attribute_value TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 4
)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/noshard/posthog.trace_attributes', '{replica}-{shard}')
PARTITION BY toDate(original_expiry_time_bucket)
ORDER BY (team_id, attribute_type, time_bucket, resource_fingerprint, attribute_key, attribute_value)
TTL original_expiry_time_bucket
SETTINGS
    index_granularity = 8192

CREATE TABLE IF NOT EXISTS posthog_test.trace_attributes2
(
    `team_id` Int32,
    `original_expiry_time_bucket` DateTime64(0),
    `time_bucket` DateTime64(0),
    `service_name` LowCardinality(String),
    `resource_fingerprint` UInt64 DEFAULT 0,
    `attribute_key` LowCardinality(String),
    `attribute_value` String,
    `attribute_type` LowCardinality(String),
    `attribute_count` SimpleAggregateFunction(sum, UInt64),
    INDEX idx_attribute_key attribute_key TYPE bloom_filter(0.01) GRANULARITY 4,
    INDEX idx_attribute_value attribute_value TYPE bloom_filter(0.01) GRANULARITY 4,
    INDEX idx_attribute_key_n3 attribute_key TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 4,
    INDEX idx_attribute_value_n3 attribute_value TYPE ngrambf_v1(3, 32768, 3, 0) GRANULARITY 4
)
ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/noshard/posthog.trace_attributes2', '{replica}-{shard}')
PARTITION BY toDate(original_expiry_time_bucket)
ORDER BY (team_id, attribute_type, time_bucket, resource_fingerprint, attribute_key, attribute_value)
TTL original_expiry_time_bucket
SETTINGS
    index_granularity = 8192

CREATE TABLE IF NOT EXISTS posthog_test.trace_spans_kafka_metrics
(
    `_partition` UInt32,
    `_topic` String,
    `max_of

…truncated. See the full SQL in the workflow logs.

❌ Hobby preview — Health check did not pass within 35 minutes

Failing fast because: Health check did not pass within 35 minutes

_{Run 29502862254 | Consecutive failures: 2}

trunk-io · 2026-07-14T14:15:20Z

_{View Full Report ↗︎ ⋅ Docs}

stamphog · 2026-07-15T09:45:43Z

Note

🤖 stamphog reviewed 1c16d8d73acd685ae1467cddb36c0f39b27f12a1 — verdict: REFUSED

Gates denied this PR outright (deny-listed migration files, over size ceiling, classified T2-never), and it introduces a new ClickHouse migration plus rewritten SQL table definitions on the logs cluster — risky infra/data territory needing human review, and two bot reviewers still show in-flight (eyes) reactions.

Deterministic gates DENIED: deny-list match on migrations, size over ceiling (3426L/15F substantive vs 800L cap), classified T2-never
New ClickHouse migration and rewritten SQL table definitions on the logs cluster carry data-model/infra risk needing domain review
@greptile-apps[bot] and @hex-security-app[bot] both show eyes reactions indicating in-flight reviews not yet completed

Gate mechanics and policy version

Gate		Result
prerequisites	✓	all clear
deny-list	✗	matches: migrations
size	✗	too large for auto-review (3426L, 15F substantive, 3800L/17F incl. docs/generated/snapshots — ceiling is 800L)
tier	✗	classified as T2-never: T2-never (3800L, 17F, two-areas, feat)
stamphog 2.0.0b3		`.stamphog/policy.yml` @ `6bc5119` · reviewed head `1c16d8d`

greptile-apps · 2026-07-15T09:47:11Z

_{Reviews (1): Last reviewed commit: "fix(llma): model trace tables in local l..." | Re-trigger Greptile}

hex-security-app · 2026-07-15T09:47:39Z

Reviewed the ClickHouse migration, trace table/MV definitions, named Kafka collection configuration, and the trace ingestion/authentication flow. I found no security issues to report.

veria-ai · 2026-07-15T10:00:26Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Add an idempotent migration creating the tracing product ClickHouse tables on the logs cluster: trace_spans, trace_attributes (plain, matches current prod) and trace_attributes2 (aggregating), the shared distributed tables, the span attribute materialized views (existing + v2 feeding trace_attributes2), the Kafka ingestion table + MV, and the kafka-metrics table + MV. Every statement uses CREATE ... IF NOT EXISTS so it is a no-op against the objects already created manually in prod. Base and distributed tables are registered in schema.py for local/CI parity; the materialized views and Kafka table stay migration-only, mirroring the logs cluster convention. Adds the warpstream_traces named collection to local ClickHouse config for the Kafka table. Generated-By: PostHog Code Task-Id: b927db46-acf2-4a29-9c9d-59bf46d38cb6

The clickhouse smoke test (per-cluster topology) runs migrations against the multinode ClickHouse config, not the local dev default.xml. The kafka_trace_spans_avro table uses the warpstream_traces named collection, which was only added to default.xml, so the migration failed on the logs node with NAMED_COLLECTION_DOESNT_EXIST. Add it to every multinode node config alongside the existing warpstream_logs entry. Generated-By: PostHog Code Task-Id: b927db46-acf2-4a29-9c9d-59bf46d38cb6

The multinode migration smoke test introspects the migrated local logs node and diffs it against posthog/clickhouse/hcl/golden/local-logs.hcl. The new tracing migration creates 15 objects on the logs node that the HCL layer never modeled, so check-live reported drift and failed. Add the trace objects (trace_spans, trace_attributes, trace_attributes2, the distributed tables, the span/resource/name attribute MVs and their v2 variants, the Kafka ingestion table + MV, and the kafka-metrics table + MV) to roles/logs/local and regenerate the golden and build-from-scratch SQL via hclexp. Scoped to the local logs node, which is what the smoke test checks. Generated-By: PostHog Code Task-Id: b927db46-acf2-4a29-9c9d-59bf46d38cb6

The kafka_trace_spans_avro_mv projected everything except an original_expiry_timestamp, so inserted spans took the trace_spans column default (epoch). With PARTITION BY toDate(original_expiry_timestamp) and TTL original_expiry_timestamp, those rows land in a 1970 partition that TTL immediately drops. Derive it from observed_timestamp plus the retention-days header (default 15), matching the logs ingestion mv. Regenerate the local logs HCL golden and build-from-scratch SQL for the updated mv. Generated-By: PostHog Code Task-Id: b927db46-acf2-4a29-9c9d-59bf46d38cb6

Add is_root_span as a dimension to projection_aggregate_counts. The Spans-view sparkline keeps summing event_count per minute across the new dimension; the Traces-view sparkline (distinct traces per minute) can now serve from the projection via sumIf(event_count, is_root_span = 1) — one root span per trace — instead of the uniqExactIf(trace_id, ...) raw scan. Regenerate the schema snapshot and the local logs HCL golden and SQL. Generated-By: PostHog Code Task-Id: b927db46-acf2-4a29-9c9d-59bf46d38cb6

_bytes_uncompressed and _bytes_compressed used toInt64OrNull on the Kafka byte-size headers, but the trace_spans target columns are non-nullable UInt64. A missing or malformed header yields NULL, and inserting NULL into a non-nullable column fails the materialized-view insert (CANNOT_INSERT_NULL_IN_ORDINARY_COLUMN), dropping otherwise valid spans. Default to 0 via toInt64OrDefault, matching the _record_count handling on the line above. Regenerate the local logs HCL golden and SQL for the updated mv. Generated-By: PostHog Code Task-Id: b927db46-acf2-4a29-9c9d-59bf46d38cb6

Master added a duplicate-object guard (duplicates-baseline.txt) and renamed the local logs env to local-multi. Modeling the trace objects in the self-contained roles/logs/local layer makes trace_spans_kafka_metrics and trace_spans_to_kafka_metrics_mv appear in two composed layers (local plus prod-us), which the guard flags. The self-contained local layer re-declaring cloud objects is intentional — the same reason the other trace objects are already baselined — so add these two to the baseline. Generated-By: PostHog Code Task-Id: b927db46-acf2-4a29-9c9d-59bf46d38cb6

hex-security-app · 2026-07-16T13:36:59Z

+        toStartOfInterval(timestamp, toIntervalMinute(10)) AS time_bucket,
        service_name AS service_name,
        resource_fingerprint,
        arrayJoin(resource_attributes) AS attribute,


Bound resource-attribute fanout

resource_attributes originates from the authenticated OTLP payload and has no count or key/value-length cap before arrayJoin. A sender can put a large resource map once in a ≤2 MB request and attach it to many spans; capture-logs duplicates that map into every span, while this view and its …2 counterpart each emit one indexed attribute row per key per span. Quota and rate limiting charge only the original request bytes, so a project-token holder can turn a small accepted payload into millions of ClickHouse writes and exhaust the shared logs cluster.

Prompt To Fix With AI

Enforce trace resource-attribute limits at ingestion before rows are created: cap the number of resource attributes and the encoded key/value sizes (and ideally cap spans per request). Apply equivalent validation/filtering in the ClickHouse resource-attribute materialized views as defense in depth, including both trace_attributes destinations. Ensure quota/rate-limit accounting reflects the bounded payload that will be expanded.

_{Severity: medium | Confidence: 94% | React with 👍 if useful or 👎 if not}

deployment-status-posthog · 2026-07-16T15:52:31Z

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-07-16 15:52 UTC	Run
prod-us	✅ Deployed	2026-07-16 16:08 UTC	Run
prod-eu	✅ Deployed	2026-07-16 16:07 UTC	Run

frankh self-assigned this Jul 14, 2026

frankh requested a review from a team July 15, 2026 09:37

frankh added the stamphog Request AI approval (no full review) label Jul 15, 2026

frankh marked this pull request as ready for review July 15, 2026 09:44

frankh requested a review from jonmcwest July 15, 2026 09:44

stamphog Bot removed the stamphog Request AI approval (no full review) label Jul 15, 2026

greptile-apps Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread posthog/clickhouse/traces/spans.py

Comment thread posthog/clickhouse/traces/spans.py Outdated

veria-ai Bot reviewed Jul 15, 2026

View reviewed changes

Comment thread posthog/clickhouse/traces/spans.py

frankh requested review from pawel-cebula and rorylshanks July 16, 2026 10:06

orian approved these changes Jul 16, 2026

View reviewed changes

frankh force-pushed the posthog-code/trace-clickhouse-tables branch from 7a1efd8 to da9c834 Compare July 16, 2026 13:02

graphite-app Bot reviewed Jul 16, 2026

View reviewed changes

Comment thread posthog/clickhouse/traces/spans.py Outdated

rorylshanks approved these changes Jul 16, 2026

View reviewed changes

frankh added 7 commits July 16, 2026 14:28

frankh force-pushed the posthog-code/trace-clickhouse-tables branch from 419ac6f to 8b98fd7 Compare July 16, 2026 13:35

hex-security-app Bot reviewed Jul 16, 2026

View reviewed changes

frankh merged commit e794f24 into master Jul 16, 2026
250 of 251 checks passed

frankh deleted the posthog-code/trace-clickhouse-tables branch July 16, 2026 14:26

mendral-app Bot mentioned this pull request Jul 17, 2026

fix(test): Fix test INSERT column order #71819

Open

rnegron mentioned this pull request Jul 17, 2026

fix(tracing): keep attribute fixtures inside TTL #71966

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(llma): add clickhouse migration for tracing tables - #70737

feat(llma): add clickhouse migration for tracing tables#70737
frankh merged 7 commits into
masterfrom
posthog-code/trace-clickhouse-tables

frankh commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026 •

edited

Loading

ClickHouse migration SQL per cloud environment

Uh oh!

trunk-io Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

stamphog Bot commented Jul 15, 2026

Uh oh!

greptile-apps Bot commented Jul 15, 2026

Uh oh!

Uh oh!

Uh oh!

hex-security-app Bot commented Jul 15, 2026

Uh oh!

Uh oh!

veria-ai Bot commented Jul 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

hex-security-app Bot Jul 16, 2026

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

frankh commented Jul 14, 2026

Problem

Changes

How did you test this code?

Docs update

🤖 Agent context

Uh oh!

github-actions Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 CI report

ClickHouse migration SQL per cloud environment

Uh oh!

trunk-io Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stamphog Bot commented Jul 15, 2026

Uh oh!

greptile-apps Bot commented Jul 15, 2026

Uh oh!

Uh oh!

Uh oh!

hex-security-app Bot commented Jul 15, 2026

Uh oh!

Uh oh!

veria-ai Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

Security review

Uh oh!

Uh oh!

hex-security-app Bot Jul 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deployment-status-posthog Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploy status

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jul 14, 2026 •

edited

Loading

trunk-io Bot commented Jul 14, 2026 •

edited

Loading

veria-ai Bot commented Jul 15, 2026 •

edited

Loading

deployment-status-posthog Bot commented Jul 16, 2026 •

edited

Loading