fix: Strip toTimeZone from timestamp comparison in WHERE clause to fix primary key usage

[robbie-c]

TLDR: 22-37% faster 7-day funnel, 62% fewer granules scanned — by restoring ClickHouse partition/PK pruning. Use toTimeZone broke clickhouse's ability to use the primary key, this fixes that.

We found this with pi-autoresearch in #hackathon-lisbon-query-performance-autoresearch

PostHog Code continues:

Problem

The events table uses PARTITION BY toYYYYMM(timestamp) and ORDER BY (team_id, toDate(timestamp), event, ...), but HogQL wraps all timestamp fields with toTimeZone(timestamp, tz) for timezone support. ClickHouse's query planner can't derive toYYYYMM(timestamp) or toDate(timestamp) bounds from toTimeZone(timestamp, tz) comparisons, so both partition pruning and primary key usage were lost.

The MinMax index on timestamp still worked (it evaluates toTimeZone() against per-part min/max values), which is why queries weren't catastrophically slow — but after MinMax narrowed down the parts, ClickHouse scanned all granules within those parts instead of using toDate(timestamp) in the primary key to skip to the right date range.

Verified via EXPLAIN PLAN indexes=1, json=1 on sharded_events with load_balancing='in_order':

• Without fix: Partition: Condition='true' (not pruning), PK keys ['team_id', 'event'] → 60,683 granules after skip index
• With fix: Partition: toYYYYMM(timestamp) in [202604, 202604], PK keys ['team_id', 'toDate(timestamp)', 'event'] → 23,291 granules after skip index (62% reduction)

Changes

In PropertySwapper.visit_compare_operation, for top-level WHERE range comparisons (>=, >, <, <=), we move the timezone from the field side to the constant side:

-- Before (blocks partition/PK pruning)
toTimeZone(timestamp, 'US/Pacific') >= '2024-03-01'

-- After (planner sees bare timestamp, can prune)
timestamp >= toDateTime64('2024-03-01', 6, 'US/Pacific')

This works because toTimeZone() only changes display metadata on DateTime values — the underlying epoch is unchanged. By annotating the constant with the timezone instead (toDateTime64(constant, 6, tz)), ClickHouse interprets it in the correct timezone while the planner can see the bare timestamp field for pruning.

Key design decisions:

• Only applies to top-level range comparisons — not inside function calls like if(), coalesce() — tracked via _inside_call_depth
• toTimeZone is preserved in SELECT expressions for correct display
• Constants already carrying timezone (3-arg toDateTime64) are left unchanged
• Wrapper functions like assumeNotNull(toDateTime(...)) are recursed into

We also compared against an indexHint() approach (adding indexHint(bare_timestamp >= constant ± tz_offset) alongside the toTimeZone comparison). The stripping approach outperforms it significantly — cleaner planner signal, fewer granules, lower variance.

Credit to @andyzzhao for the insight that we can strip toTimeZone entirely and annotate the constant instead.

How did you test this code?

New tests in TestTimezoneIndexPruning (in test_property_types.py):

EXPLAIN-based pruning tests:

• test_bare_timestamp_prunes_partition_and_primary_key — baseline confirming bare timestamp pruning works
• test_toTimeZone_breaks_partition_and_pk_pruning — canary test asserting the ClickHouse limitation still exists (if it starts failing, the workaround is removable)
• test_hogql_compiled_query_has_partition_pruning — full HogQL pipeline now has partition + PK pruning

Correctness tests (real ClickHouse queries with event data):

• test_dst_boundary_does_not_drop_events — America/New_York DST transition
• test_positive_utc_offset_does_not_drop_events — Asia/Tokyo (UTC+9)
• test_utc_returns_correct_results
• test_brazil_historical_dst_does_not_drop_events — historical DST that was later abolished
• test_half_hour_offset_does_not_drop_events — Asia/Kolkata (UTC+5:30)
• test_lord_howe_half_hour_dst_does_not_drop_events — 30-minute DST shift

SQL shape tests:

• test_toTimeZone_stripped_from_where_but_kept_in_select
• test_toTimeZone_not_stripped_inside_function_calls
• test_constant_gets_timezone_annotation

Benchmarks

Profiled on prod ClickHouse via Metabase — 7-day funnel (pageview → autocapture) on team 2, US/Pacific timezone. All variants used load_balancing='in_order' for consistency. Each variant run 5 times.

EXPLAIN granule counts (single shard via sharded_events):

Stage	Baseline	indexHint +7/+8	Stripping (this PR)
Partition	true ❌	202604 ✅	202604 ✅
PK keys	team_id, event	team_id, toDate(ts), event	team_id, toDate(ts), event
PK granules	73,638	60,434	23,828
Skip granules	60,683	55,716	23,291

Execution timing (distributed events table, 5 runs each):

Metric	Baseline	indexHint +7/+8	Stripping (this PR)
Best of 5	2,824ms	4,127ms	2,192ms (-22%)
Trimmed mean (mid 3)	4,694ms	7,236ms	2,954ms (-37%)
All durations	2824, 3357, 4755, 5970, 6440	4127, 6582, 6677, 8449, 9307	2192, 2464, 3149, 3250, 8356

Publish to changelog?

No

Docs update

No docs changes needed.

🤖 LLM context

Authored by PostHog Code. The approach evolved during the session:

1. Started with indexHint() + bare timestamp bounds with a 14-hour buffer
2. Moved to timezone-specific buffers (min/max offsets from a generated lookup table)
3. Realized the complexity wasn't worth it — Andy suggested simply stripping toTimeZone and annotating the constant instead, which is both simpler and more effective (fewer granules, lower variance, no buffer arithmetic)

---

Created with PostHog Code

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to 8cde337 while workflow was testing c6daaa7.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to 3baed27 while workflow was testing 8cde337.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to 4a29517 while workflow was testing 3baed27.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[github-actions]

🎭 Playwright report · View test results →

⚠️ 4 flaky tests:

• Logout in another tab results in logout in the current tab too (chromium)
• create experiment via wizard, add metrics, and launch (chromium)
• Save view (chromium)
• Materialize view pane (chromium)

These issues are not necessarily caused by your changes.
Annoyed by this comment? Help fix flakies and failures and it'll disappear!

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to 45ee5e0 while workflow was testing 4a29517.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to 2191322 while workflow was testing 45ee5e0.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[greptile-apps]

Prompt To Fix All With AI

This is a comment left during a code review.
Path: posthog/hogql/transforms/property_types.py
Line: 195-200

Comment:
**Imprecise type annotation on `_RANGE_OPS`**

`set[str]` should be `set[ast.CompareOperationOp]`. Even though `CompareOperationOp` is a `StrEnum` (and its members technically _are_ strings), the declared type doesn't communicate intent and the membership check on line 233 (`result.op not in self._RANGE_OPS`) is then typed as comparing a `CompareOperationOp` against a `set[str]`.

```suggestion
    _RANGE_OPS: set[ast.CompareOperationOp] = {
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: posthog/hogql/transforms/test/test_property_types.py
Line: 325-366

Comment:
**Use `@parameterized.expand` instead of `subTest`**

The repo convention (AGENTS.md: "We always prefer parameterised tests") calls for the `parameterized` library rather than `self.subTest`. `test_toTimeZone_breaks_partition_and_pk_pruning` loops with `subTest`, and `test_index_hint_not_added_inside_function_calls` tests three separate SQL/count combinations in sequence. Both are candidates for `@parameterized.expand`.

For example, `test_index_hint_not_added_inside_function_calls` could become:

```python
@parameterized.expand([
    ("if_in_select",
     "SELECT if(timestamp >= '2024-03-01', 'yes', 'no') FROM events",
     0),
    ("mixed_where_and_if",
     "SELECT if(timestamp >= '2024-01-01', 'new', 'old') FROM events "
     "WHERE timestamp >= '2024-03-01' AND timestamp < '2024-04-01'",
     2),
    ("deep_nesting",
     "SELECT coalesce(if(timestamp < '2024-06-01', NULL, timestamp), now()) FROM events",
     0),
])
def test_index_hint_not_added_inside_function_calls(self, _name, hogql, expected_count):
    sql, _ = self._compile_hogql(hogql, timezone="America/New_York")
    assert sql.count("indexHint") == expected_count
```

**Context Used:** Do not attempt to comment on incorrect alphabetica... ([source](https://app.greptile.com/review/custom-context?memory=instruction-0))

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: posthog/hogql/transforms/test/test_property_types.py
Line: 499-520

Comment:
**Misleading test name contradicts its own docstring**

The method is named `test_utc_zero_buffer_returns_correct_results`, but the docstring immediately says _"UTC gets the same 14h buffer as other timezones."_ The name implies no buffer is applied for UTC, which is the opposite of what the code does (`_tz_buffer_hours = 14` whenever `setTimeZones=True`, regardless of timezone). Consider renaming to something like `test_utc_timezone_returns_correct_results`.

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "fix: add indexHint to restore partition/..." | Re-trigger Greptile}

[tests-posthog]

Query snapshots: Backend query snapshots updated

Changes: 62 snapshots (62 modified, 0 added, 0 deleted)

What this means:

• Query snapshots have been automatically updated to match current output
• These changes reflect modifications to database queries or schema

Next steps:

• Review the query changes to ensure they're intentional
• If unexpected, investigate what caused the query to change

Review snapshot changes →

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to f901a92 while workflow was testing 6ff37d0.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[robbie-c]

Raw benchmark data (7-day funnel, run 9)

All runs used load_balancing='in_order' for replica consistency. EXPLAIN plans queried against sharded_events (single shard). Execution runs against distributed events table.

EXPLAIN indexes (sharded_events, load_balancing=in_order)

Baseline indexes

[
  {
    "Type": "MinMax",
    "Keys": [
      "timestamp"
    ],
    "Condition": "and((toTimezone(timestamp, 'US/Pacific') in (-Inf, '1776409199.999999']), (toTimezone(timestamp, 'US/Pacific') in ['1775718000', +Inf)))",
    "Initial Parts": 1810,
    "Selected Parts": 49,
    "Initial Granules": 189403366,
    "Selected Granules": 6194548
  },
  {
    "Type": "Partition",
    "Condition": "true",
    "Initial Parts": 49,
    "Selected Parts": 49,
    "Initial Granules": 6194548,
    "Selected Granules": 6194548
  },
  {
    "Type": "PrimaryKey",
    "Keys": [
      "team_id",
      "event"
    ],
    "Condition": "and((event in 2-element set), (team_id in [2, 2]))",
    "Search Algorithm": "generic exclusion search",
    "Initial Parts": 49,
    "Selected Parts": 49,
    "Initial Granules": 6194548,
    "Selected Granules": 73638
  },
  {
    "Type": "Skip",
    "Name": "minmax_sharded_events_timestamp",
    "Description": "minmax GRANULARITY 1",
    "Initial Parts": 49,
    "Selected Parts": 47,
    "Initial Granules": 73638,
    "Selected Granules": 60683
  }
]

indexHint +7/+8 indexes

[
  {
    "Type": "MinMax",
    "Keys": [
      "timestamp"
    ],
    "Condition": "and((timestamp in (-Inf, '1776437999.999999']), and((toTimezone(timestamp, 'US/Pacific') in (-Inf, '1776409199.999999']), and((timestamp in ['1775743200', +Inf)), (toTimezone(timestamp, 'US/Pacific') in ['1775718000', +Inf)))))",
    "Initial Parts": 1863,
    "Selected Parts": 51,
    "Initial Granules": 191551084,
    "Selected Granules": 5514013
  },
  {
    "Type": "Partition",
    "Keys": [
      "toYYYYMM(timestamp)"
    ],
    "Condition": "and((toYYYYMM(timestamp) in (-Inf, 202604]), (toYYYYMM(timestamp) in [202604, +Inf)))",
    "Initial Parts": 51,
    "Selected Parts": 51,
    "Initial Granules": 5514013,
    "Selected Granules": 5514013
  },
  {
    "Type": "PrimaryKey",
    "Keys": [
      "team_id",
      "toDate(timestamp)",
      "event"
    ],
    "Condition": "and((event in 2-element set), and((toDate(timestamp) in (-Inf, 20560]), and((toDate(timestamp) in [20552, +Inf)), (team_id in [2, 2]))))",
    "Search Algorithm": "generic exclusion search",
    "Initial Parts": 51,
    "Selected Parts": 51,
    "Initial Granules": 5514013,
    "Selected Granules": 60434
  },
  {
    "Type": "Skip",
    "Name": "minmax_sharded_events_timestamp",
    "Description": "minmax GRANULARITY 1",
    "Initial Parts": 51,
    "Selected Parts": 49,
    "Initial Granules": 60434,
    "Selected Granules": 55716
  }
]

Stripping indexes (this PR)

[
  {
    "Type": "MinMax",
    "Keys": [
      "timestamp"
    ],
    "Condition": "and((timestamp in (-Inf, '1776409199.999999']), (timestamp in ['1775718000', +Inf)))",
    "Initial Parts": 1990,
    "Selected Parts": 60,
    "Initial Granules": 180809332,
    "Selected Granules": 5177243
  },
  {
    "Type": "Partition",
    "Keys": [
      "toYYYYMM(timestamp)"
    ],
    "Condition": "and((toYYYYMM(timestamp) in (-Inf, 202604]), (toYYYYMM(timestamp) in [202604, +Inf)))",
    "Initial Parts": 60,
    "Selected Parts": 60,
    "Initial Granules": 5177243,
    "Selected Granules": 5177243
  },
  {
    "Type": "PrimaryKey",
    "Keys": [
      "team_id",
      "toDate(timestamp)",
      "event"
    ],
    "Condition": "and((event in 2-element set), and((toDate(timestamp) in (-Inf, 20560]), and((toDate(timestamp) in [20552, +Inf)), (team_id in [2, 2]))))",
    "Search Algorithm": "generic exclusion search",
    "Initial Parts": 60,
    "Selected Parts": 60,
    "Initial Granules": 5177243,
    "Selected Granules": 23828
  },
  {
    "Type": "Skip",
    "Name": "minmax_sharded_events_timestamp",
    "Description": "minmax GRANULARITY 1",
    "Initial Parts": 60,
    "Selected Parts": 56,
    "Initial Granules": 23828,
    "Selected Granules": 23291
  }
]

Execution results (query_log, 5 runs each)

Query log results

[
  {
    "variant": "baseline",
    "query_duration_ms": "4,755",
    "read_rows": "27,535,611",
    "read_bytes_human": "2.24 GiB",
    "selected_parts": "59",
    "selected_marks": "62,117"
  },
  {
    "variant": "baseline",
    "query_duration_ms": "5,970",
    "read_rows": "25,203,657",
    "read_bytes_human": "2.23 GiB",
    "selected_parts": "73",
    "selected_marks": "20,975"
  },
  {
    "variant": "baseline",
    "query_duration_ms": "2,824",
    "read_rows": "27,530,511",
    "read_bytes_human": "2.24 GiB",
    "selected_parts": "73",
    "selected_marks": "23,829"
  },
  {
    "variant": "baseline",
    "query_duration_ms": "6,440",
    "read_rows": "25,745,832",
    "read_bytes_human": "2.23 GiB",
    "selected_parts": "67",
    "selected_marks": "61,636"
  },
  {
    "variant": "baseline",
    "query_duration_ms": "3,357",
    "read_rows": "27,649,289",
    "read_bytes_human": "2.24 GiB",
    "selected_parts": "81",
    "selected_marks": "24,708"
  },
  {
    "variant": "indexHint +7/+8",
    "query_duration_ms": "9,307",
    "read_rows": "25,427,025",
    "read_bytes_human": "2.13 GiB",
    "selected_parts": "81",
    "selected_marks": "19,630"
  },
  {
    "variant": "indexHint +7/+8",
    "query_duration_ms": "6,582",
    "read_rows": "26,139,681",
    "read_bytes_human": "2.13 GiB",
    "selected_parts": "47",
    "selected_marks": "23,470"
  },
  {
    "variant": "indexHint +7/+8",
    "query_duration_ms": "4,127",
    "read_rows": "23,915,002",
    "read_bytes_human": "2.13 GiB",
    "selected_parts": "62",
    "selected_marks": "23,085"
  },
  {
    "variant": "indexHint +7/+8",
    "query_duration_ms": "6,677",
    "read_rows": "25,213,166",
    "read_bytes_human": "2.13 GiB",
    "selected_parts": "95",
    "selected_marks": "19,674"
  },
  {
    "variant": "indexHint +7/+8",
    "query_duration_ms": "8,449",
    "read_rows": "26,260,186",
    "read_bytes_human": "2.13 GiB",
    "selected_parts": "83",
    "selected_marks": "59,663"
  },
  {
    "variant": "stripping",
    "query_duration_ms": "2,192",
    "read_rows": "27,488,851",
    "read_bytes_human": "2.23 GiB",
    "selected_parts": "55",
    "selected_marks": "24,693"
  },
  {
    "variant": "stripping",
    "query_duration_ms": "3,149",
    "read_rows": "27,469,271",
    "read_bytes_human": "2.23 GiB",
    "selected_parts": "57",
    "selected_marks": "61,281"
  },
  {
    "variant": "stripping",
    "query_duration_ms": "3,250",
    "read_rows": "27,001,413",
    "read_bytes_human": "2.23 GiB",
    "selected_parts": "80",
    "selected_marks": "21,475"
  },
  {
    "variant": "stripping",
    "query_duration_ms": "8,356",
    "read_rows": "27,260,208",
    "read_bytes_human": "2.24 GiB",
    "selected_parts": "103",
    "selected_marks": "21,513"
  },
  {
    "variant": "stripping",
    "query_duration_ms": "2,464",
    "read_rows": "26,727,215",
    "read_bytes_human": "2.23 GiB",
    "selected_parts": "77",
    "selected_marks": "24,968"
  }
]

[tests-posthog]

Query snapshots: Backend query snapshots updated

Changes: 68 snapshots (68 modified, 0 added, 0 deleted)

What this means:

• Query snapshots have been automatically updated to match current output
• These changes reflect modifications to database queries or schema

Next steps:

• Review the query changes to ensure they're intentional
• If unexpected, investigate what caused the query to change

Review snapshot changes →

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to 997544e while workflow was testing 7e571fb.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[tests-posthog]

⏭️ Skipped snapshot commit because branch advanced to e074dd8 while workflow was testing 997544e.

The new commit will trigger its own snapshot update workflow.

If you expected this workflow to succeed: This can happen due to concurrent commits. To get a fresh workflow run, either:

• Merge master into your branch, or
• Push an empty commit: git commit --allow-empty -m 'trigger CI' && git push

[tests-posthog]

Query snapshots: Backend query snapshots updated

Changes: 68 snapshots (68 modified, 0 added, 0 deleted)

What this means:

• Query snapshots have been automatically updated to match current output
• These changes reflect modifications to database queries or schema

Next steps:

• Review the query changes to ensure they're intentional
• If unexpected, investigate what caused the query to change

Review snapshot changes →

[deployment-status-posthog]

Deploy status

Environment	Status	Deployed At	Workflow
dev	✅ Deployed	2026-04-16 16:25 UTC	Run
prod-us	✅ Deployed	2026-04-16 16:39 UTC	Run
prod-eu	✅ Deployed	2026-04-16 16:45 UTC	Run