Skip to content

fix: add hogql anti-pattern rules to mcp sql guidance - #70536

Merged
andyzzhao merged 5 commits into
masterfrom
andy/mcp-sql-dot-access-guidance
Jul 13, 2026
Merged

fix: add hogql anti-pattern rules to mcp sql guidance#70536
andyzzhao merged 5 commits into
masterfrom
andy/mcp-sql-dot-access-guidance

Conversation

@andyzzhao

@andyzzhao andyzzhao commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Problem

LLMs can write poor-performing HogQL. I created an LLM judge evaluation to find anti-patterns in LLM-written HogQL queries.

Changes

products/posthog_ai/skills/querying-posthog-data/references/guidelines.md. It's bundled into the MCP server, so the rules land in both the execute-sql tool description and the querying skill.

  • Don't hunt for event or property names with LIKE '%term%'. Look up the real names with read-data-schema and match them exactly. A wildcard search reads every row in the time range; an exact match lets ClickHouse skip most of the table.
  • Every events query needs a time filter in its WHERE clause. A time condition inside countIf(...) doesn't count, and a selective-looking id filter (session, trace, user) doesn't either: without a time filter the whole history gets scanned.
  • Read the events table once per question where possible, using conditional aggregates like countIf instead of self-joins or repeating the same subquery. A WITH subquery used twice runs twice.
  • Count unique users with uniq(person_id), not uniq(distinct_id). This one is about correctness than speed
  • Don't GROUP BY values with unlimited variety (raw URLs, ids, free text) over long time ranges. The database holds every distinct value in memory, and LIMIT doesn't help.

Left alone on purpose: the uniq(distinct_id) examples in models-mcp.md, which mirror backend code and belong to the MCP analytics team.

How did you test this code?

Benchmarked the two biggest rewrites on our test ClickHouse cluster:

  • Finding events with "insight" in the name: the wildcard version read 23x more rows (1.3B vs 56M) and 12x more data than matching the 48 exact names, for the identical result.
  • Grouping by raw URL vs by URL with ids stripped out: same data read, but 6x more memory (2.9 GiB vs 495 MiB).

After this merges, the same evaluations that produced the fail rates above tell us whether the rules actually changed client behavior.

Automatic notifications

  • Publish to changelog?
  • Alert Sales and Marketing teams?

Docs update

Not needed: this PR is itself the guidance change.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Written by Claude Code, directed by the assignee, as part of a quality loop: capture every MCP SQL query with its stated intent, score them with online evaluations, fix the guidance where the scores show problems, then watch the rates move. The rules come from the internal optimizing-clickhouse-and-hogql-queries skill plus the benchmarks above. Started as a single JSONExtract rule, expanded to the full measured set, then the JSONExtract rule was dropped on the assignee's direction after we confirmed the HogQL engine already rewrites the common form to the materialized column. No repo skills were invoked for the edits themselves (markdown-only change).

The execute-sql querying guidance never told MCP clients how property
access actually works: properties.foo is resolved by the HogQL printer
to a materialized column when one exists (up to ~100x cheaper) and
falls back to JSON extraction itself when none does, so hand-written
JSONExtract*(properties, ...) can only tie or lose. An online
evaluation on execute-sql generations measures the hand-written form
at roughly 2% of MCP-authored queries.

Adds the dot-access rule (with typed-read casts and a carve-out for
legitimate JSONExtract on non-properties JSON columns) to the property
access section and the performance rules of guidelines.md, which feeds
both the querying-posthog-data skill and the execute-sql tool
description.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHNqSqPGjvFfgv6ahQn7hQ
@andyzzhao andyzzhao self-assigned this Jul 13, 2026
Extends the querying guidance with the remaining measured anti-patterns
from the execute-sql online evaluations:

- Search types: substring LIKE/ILIKE on events strings is a full scan;
  discover real values via read-data-schema and use equality or an
  anchored prefix (largest failure mode, ~20% of MCP-authored queries).
- Time ranges: the bound must be a WHERE predicate on timestamp
  (aggregate-argument conditions filter nothing), and id-anchored point
  lookups still need one (dominant class of judge failures).
- JOINs: scan events once with conditional aggregates instead of
  self-joins, repeated same-range subqueries, or twice-referenced CTEs.
- Other rules: uniq(person_id) not uniq(distinct_id); avoid unbounded
  high-cardinality GROUP BY over wide windows.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHNqSqPGjvFfgv6ahQn7hQ
@andyzzhao andyzzhao changed the title fix(phai): teach dot access over JSONExtract in sql querying guidance fix(phai): performance rules in mcp sql querying guidance Jul 13, 2026
@andyzzhao andyzzhao changed the title fix(phai): performance rules in mcp sql querying guidance fix: teach dot access over JSONExtract in sql querying guidance Jul 13, 2026
Benchmarked both on the Test Cluster: uniq(person_id) is a correctness
rule (distinct_id overcounts users; perf roughly neutral since person
resolution adds the overrides join), and the high-cardinality GROUP BY
rule pays on the memory axis (6x peak memory on identical scans), not
latency. Label them accordingly so the performance framing stays honest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHNqSqPGjvFfgv6ahQn7hQ
@andyzzhao
andyzzhao marked this pull request as ready for review July 13, 2026 20:57
@pr-assigner-resolver-posthog
pr-assigner-resolver-posthog Bot requested a review from a team July 13, 2026 20:57
@greptile-apps

greptile-apps Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Reviews (1): Last reviewed commit: "fix(phai): label person_id rule correctn..." | Re-trigger Greptile

Comment thread products/posthog_ai/skills/querying-posthog-data/references/guidelines.md Outdated
Comment thread products/posthog_ai/skills/querying-posthog-data/references/guidelines.md Outdated
Comment thread products/posthog_ai/skills/querying-posthog-data/references/guidelines.md Outdated
Comment thread products/posthog_ai/skills/querying-posthog-data/references/guidelines.md Outdated
@andyzzhao
andyzzhao marked this pull request as draft July 13, 2026 22:03
The HogQL engine already rewrites JSONExtractString(properties, 'x')
to the materialized column when one exists, the never-rescued variants
measure at only ~1% of MCP-authored queries, and the remaining forms
(Raw on nested JSON, key listing, existence checks) have legitimate
uses that a blanket rule would suppress. Keep the guidance focused on
the patterns with the largest measured cost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHNqSqPGjvFfgv6ahQn7hQ
@andyzzhao
andyzzhao marked this pull request as ready for review July 13, 2026 22:41
@greptile-apps

greptile-apps Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Reviews (2): Last reviewed commit: "fix(phai): drop the jsonextract dot-acce..." | Re-trigger Greptile

…idelines.md

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
@andyzzhao
andyzzhao requested a review from a team July 13, 2026 22:46
@andyzzhao
andyzzhao merged commit f93c4d0 into master Jul 13, 2026
186 checks passed
@andyzzhao
andyzzhao deleted the andy/mcp-sql-dot-access-guidance branch July 13, 2026 23:43
@andyzzhao andyzzhao changed the title fix: teach dot access over JSONExtract in sql querying guidance fix(phai): add hogql anti-pattern rules to mcp sql guidance Jul 13, 2026
@andyzzhao andyzzhao changed the title fix(phai): add hogql anti-pattern rules to mcp sql guidance fix: add hogql anti-pattern rules to mcp sql guidance Jul 13, 2026
@deployment-status-posthog

deployment-status-posthog Bot commented Jul 14, 2026

Copy link
Copy Markdown

Deploy status

Environment Status Deployed At Workflow
dev ✅ Deployed 2026-07-14 02:24 UTC Run
prod-us ✅ Deployed 2026-07-14 02:36 UTC Run
prod-eu ✅ Deployed 2026-07-14 02:36 UTC Run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants