docs(gfql): withdraw the OLAP q8 magnitude claim (5.0 ms vs 1,004 ms, 200x) - #1817
Merged
Conversation
… 200x) `gfql/performance.rst` published the graph-benchmark OLAP result as "q8 runs in 5.0 ms vs 1,004 ms for embedded Kuzu (200x)". The claim is withdrawn -- reconciled, not merely re-measured. Our own side held up: 5.0 ms published vs 5.06 ms re-measured. The competitor side did not. The old harness replaced q8/q9 path enumeration with an `OPTIONAL MATCH` degree-product rewrite and fed Kuzu pre-lowercased columns built OUTSIDE the timing loop, while GFQL paid `toLower()` in-query. Re-running that same handicapped configuration reproduces 964.60 ms on the same box -- a 98.9x handicap applied to the competitor, not GFQL speed. Canonical Kuzu 0.11.3, position-balanced and row-validated at 100k persons / 2.4M FOLLOWS, is 9.76 ms vs 5.06 ms = 1.93x. The directional claim (`engine='polars'` wins q8) is measured and is kept; only the magnitude goes. No replacement figure is published here. Scope is deliberately one sentence: the q9 14.2x figure, the LadybugDB comparison figures, and the `engines.rst` "one to two orders of magnitude" clause are all left at their master text -- those are being researched/re-measured separately, and a published number is not removed on a re-measurement alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
lmeyerov
force-pushed
the
docs/withdraw-refuted-bench-numbers
branch
from
July 28, 2026 06:41
73eae42 to
b050e1e
Compare
lmeyerov
added a commit
that referenced
this pull request
Jul 28, 2026
Only CHANGELOG conflicted, and both entries are kept -- #1823's fused single-hop grouped aggregate and this PR's constant folding are disjoint lanes (#1823 serves q1-q4, this serves q5-q7), and gfql_fast_paths.py auto-merged with no overlap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YYZRXegrALuXd3NHH5evqx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Withdraws one published benchmark magnitude: the GFQL docs' OLAP claim that "q8 runs in 5.0 ms vs 1,004 ms for embedded Kuzu (200×)".
2 files, +3 / -3 (one sentence in
docs/source/gfql/performance.rst, plus CHANGELOG). Reviewable in under a minute.Why this one clears the bar: it was reconciled, not just re-measured
A published number is not removed on the strength of a re-measurement alone. This one is removed because we reproduced how the original was produced and found the discrepancy sits entirely on the competitor side:
engine='polars'q8The old harness did two things to the Kuzu side: it replaced q8/q9 path enumeration with an
OPTIONAL MATCHdegree-product rewrite, and it fed Kuzu pre-lowercased columns built outside the timing loop while GFQL paidtoLower()in-query. Re-running that same handicapped configuration reproduces 964.60 ms on the same box — so the published "200×" was a 98.9× handicap applied to the competitor, not GFQL speed.Re-measurement conditions: one host, position-balanced, result rows validated (0 mismatches), 100k persons / 2.4M FOLLOWS. Canonical ratio is 1.93×.
The edit
The directional claim (
engine='polars'wins q8) is itself measured — 5.06 vs 9.76 ms — so it stays. Only the magnitude goes. No replacement figure is published.What this PR deliberately does NOT do
An earlier revision of this branch also removed the q9 figure, the LadybugDB figures, and an
engines.rstclause. Those have all been backed out and are byte-identical to master here:~65×,~3.5–3.7×, and "1.8B-edge graph in <8 GB RAM" — kept. Changing them is contingent on re-running the Ladybug benchmarks and supplying a new measured value; that re-run is dispatched separately.engines.rst"(by one to two orders of magnitude withengine='polars')" — kept. Its removal was only ever a consequence of withdrawing q8+q9 together; with q9 restored, that basis is gone.benchmark_filter_pagerank.rst, charts, gate machinery, Sphinx extensions,conf.py,.rstcheck.cfg,.github/workflows/— all untouched.docs/source/gfql/engines.rstdoes not appear in this diff at all.performance.rstdiffers from master in exactly the 3 lines above.Verification
Docs build was run on
dgx-spark, not locally (per the standing rule that test suites and Dockerised builds run there; local baselines carry GPU-absence failures and a different polars version than the image).This matters because a docs-only PR SKIPS
test-docsin CI:test-docsneeds: python-lint-types, which is itself skipped by the docs-only path filter, and a job needing a skipped job is skipped. Green CI on this PR is therefore not evidence that the docs built. The explicit build is:./docs/ci.shon dgx-spark at this branch head — exit 0.validate-docs.sh(rstcheck): "Success! No issues detected". Sphinx:build succeededfor all three formats. GFQL doc-example audit passed; notebook structure validation and execution passed../bin/lint.sh— exit 0 (no Python changed).Rebased onto current master (
1537e46—ci(polars): parallelize the polars lane…/ #1814).Supersedes the docs-correction portion of #1813.