Skip to content

docs(gfql): withdraw the OLAP q8 magnitude claim (5.0 ms vs 1,004 ms, 200x) - #1817

Merged
lmeyerov merged 1 commit into
masterfrom
docs/withdraw-refuted-bench-numbers
Jul 28, 2026
Merged

docs(gfql): withdraw the OLAP q8 magnitude claim (5.0 ms vs 1,004 ms, 200x)#1817
lmeyerov merged 1 commit into
masterfrom
docs/withdraw-refuted-bench-numbers

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Withdraws one published benchmark magnitude: the GFQL docs' OLAP claim that "q8 runs in 5.0 ms vs 1,004 ms for embedded Kuzu (200×)".

2 files, +3 / -3 (one sentence in docs/source/gfql/performance.rst, plus CHANGELOG). Reviewable in under a minute.

Why this one clears the bar: it was reconciled, not just re-measured

A published number is not removed on the strength of a re-measurement alone. This one is removed because we reproduced how the original was produced and found the discrepancy sits entirely on the competitor side:

published re-measured verdict
GFQL engine='polars' q8 5.0 ms 5.06 ms our side held up
embedded Kuzu q8 1,004 ms 9.76 ms (Kuzu 0.11.3, canonical) published figure not canonical Kuzu

The old harness did two things to the Kuzu side: it replaced q8/q9 path enumeration with an OPTIONAL MATCH degree-product rewrite, and it fed Kuzu pre-lowercased columns built outside the timing loop while GFQL paid toLower() in-query. Re-running that same handicapped configuration reproduces 964.60 ms on the same box — so the published "200×" was a 98.9× handicap applied to the competitor, not GFQL speed.

Re-measurement conditions: one host, position-balanced, result rows validated (0 mismatches), 100k persons / 2.4M FOLLOWS. Canonical ratio is 1.93×.

The edit

 On the graph-benchmark OLAP multi-join queries at 100k-node scale with
-``engine='polars'``: **q8 runs in 5.0 ms vs 1,004 ms for embedded Kuzu (200×)**; q9 is
-**14.2×**.
+``engine='polars'``: GFQL wins q8 against embedded Kuzu; q9 is **14.2×**.

The directional claim (engine='polars' wins q8) is itself measured — 5.06 vs 9.76 ms — so it stays. Only the magnitude goes. No replacement figure is published.

What this PR deliberately does NOT do

An earlier revision of this branch also removed the q9 figure, the LadybugDB figures, and an engines.rst clause. Those have all been backed out and are byte-identical to master here:

  • q9 "14.2×" — kept. Being researched separately; the published number stays until that research produces a replacement.
  • LadybugDB ~65×, ~3.5–3.7×, and "1.8B-edge graph in <8 GB RAM" — kept. Changing them is contingent on re-running the Ladybug benchmarks and supplying a new measured value; that re-run is dispatched separately.
  • engines.rst "(by one to two orders of magnitude with engine='polars')" — kept. Its removal was only ever a consequence of withdrawing q8+q9 together; with q9 restored, that basis is gone.
  • benchmark_filter_pagerank.rst, charts, gate machinery, Sphinx extensions, conf.py, .rstcheck.cfg, .github/workflows/ — all untouched.

docs/source/gfql/engines.rst does not appear in this diff at all. performance.rst differs from master in exactly the 3 lines above.

Verification

Docs build was run on dgx-spark, not locally (per the standing rule that test suites and Dockerised builds run there; local baselines carry GPU-absence failures and a different polars version than the image).

This matters because a docs-only PR SKIPS test-docs in CI: test-docs needs: python-lint-types, which is itself skipped by the docs-only path filter, and a job needing a skipped job is skipped. Green CI on this PR is therefore not evidence that the docs built. The explicit build is:

  • ./docs/ci.sh on dgx-spark at this branch head — exit 0. validate-docs.sh (rstcheck): "Success! No issues detected". Sphinx: build succeeded for all three formats. GFQL doc-example audit passed; notebook structure validation and execution passed.
  • ./bin/lint.sh — exit 0 (no Python changed).

Rebased onto current master (1537e46ci(polars): parallelize the polars lane… / #1814).

Supersedes the docs-correction portion of #1813.

… 200x)

`gfql/performance.rst` published the graph-benchmark OLAP result as
"q8 runs in 5.0 ms vs 1,004 ms for embedded Kuzu (200x)". The claim is
withdrawn -- reconciled, not merely re-measured.

Our own side held up: 5.0 ms published vs 5.06 ms re-measured. The
competitor side did not. The old harness replaced q8/q9 path enumeration
with an `OPTIONAL MATCH` degree-product rewrite and fed Kuzu
pre-lowercased columns built OUTSIDE the timing loop, while GFQL paid
`toLower()` in-query. Re-running that same handicapped configuration
reproduces 964.60 ms on the same box -- a 98.9x handicap applied to the
competitor, not GFQL speed. Canonical Kuzu 0.11.3, position-balanced and
row-validated at 100k persons / 2.4M FOLLOWS, is 9.76 ms vs 5.06 ms =
1.93x.

The directional claim (`engine='polars'` wins q8) is measured and is
kept; only the magnitude goes. No replacement figure is published here.

Scope is deliberately one sentence: the q9 14.2x figure, the LadybugDB
comparison figures, and the `engines.rst` "one to two orders of
magnitude" clause are all left at their master text -- those are being
researched/re-measured separately, and a published number is not removed
on a re-measurement alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
@lmeyerov
lmeyerov force-pushed the docs/withdraw-refuted-bench-numbers branch from 73eae42 to b050e1e Compare July 28, 2026 06:41
@lmeyerov lmeyerov changed the title docs(gfql): withdraw three refuted/unsourced benchmark figures docs(gfql): withdraw the OLAP q8 magnitude claim (5.0 ms vs 1,004 ms, 200x) Jul 28, 2026
@lmeyerov
lmeyerov merged commit 7bc9b00 into master Jul 28, 2026
35 checks passed
@lmeyerov
lmeyerov deleted the docs/withdraw-refuted-bench-numbers branch July 28, 2026 07:16
lmeyerov added a commit that referenced this pull request Jul 28, 2026
Only CHANGELOG conflicted, and both entries are kept -- #1823's fused single-hop
grouped aggregate and this PR's constant folding are disjoint lanes (#1823 serves
q1-q4, this serves q5-q7), and gfql_fast_paths.py auto-merged with no overlap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant