perf(gfql): dense-domain proof kernel for the equal-domain two-hop count (q8) - #1844
Open
lmeyerov wants to merge 5 commits into
Open
perf(gfql): dense-domain proof kernel for the equal-domain two-hop count (q8)#1844lmeyerov wants to merge 5 commits into
lmeyerov wants to merge 5 commits into
Conversation
The equal-domain branch of the two-hop count fast path (graph-benchmark q8
class: MATCH (a {t})-[{r}]->(b {t})-[{r}]->(d {t}) RETURN count(*)) spends
its time restricting the rel-filtered edges to domain x domain -- two semi-
joins on polars, two isin masks on pandas/cuDF -- not counting. Profiled
one-shot on the 20k graph-benchmark (249k FOLLOWS edges, local CPU,
diagnosis-only): 5.3ms of the 9.3ms polars call is that restriction; the
two degree group_bys alone cost 2.1ms.
When the domain's ids form a dense integer interval [lo, hi]
(n_unique == hi - lo + 1) and both endpoint columns are integer, null-free
and bounded by [lo, hi], interval membership IS set membership: the
restriction is provably the identity and the degree product collapses to
two O(E) bincounts plus one O(N) aligned product-sum. Dense
type-partitioned ids are the idiomatic multi-table graph encoding
(offset-based global ids), so the proof -- a handful of O(E) min/max
reductions -- is a data property, not a query-shaped hack; any failed
guard declines to the existing memoized semi-join path unchanged
(out-of-domain endpoints, gapped/non-integer/null ids, empty edges, and a
memory guard for domains far wider than the edge count).
Engine-polymorphic via the index module's array helpers (numpy host for
pandas/polars/polars-gpu, cupy for cudf), same convention as the CSR
kernels. When the kernel serves, no cross-call memo is written: one-shot
and warm calls converge, removing the #1825 binding sensitivity for this
shape.
Local diagnosis-only effect (20k/100k graph-benchmark, cold binding,
values byte-identical): polars q8 9.3ms -> 1.8ms and pandas 47ms -> 27ms
at 20k; polars 52.8ms -> 31.5ms at 100k (bincount is the remaining
memory-bound floor there). The authoritative numbers must come from the
locked DGX rerun of the receipted matched q1-q9 lane.
Tests: T6 lane pin (kernel serves + memo path provably not entered), value
parity vs the forced-decline semi-join path on the same graph, offset and
negative interval shifts, every decline guard pinned, and the memo-lane
reachability pin under the new seam. The H3 memo miss/hit test's graph ids
gain a deliberate gap so it keeps pinning the memo lane it documents.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
…formance Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
This was referenced Aug 3, 2026
Open
lmeyerov
commented
Aug 3, 2026
| return int(lo), int(hi) # type: ignore[arg-type] | ||
| import numpy as np | ||
| s_pd = domain_nodes[node_col] | ||
| dtype = getattr(s_pd, "dtype", None) |
Contributor
Author
There was a problem hiding this comment.
Restructured into a fully typed polars helper _dense_interval_polars(s: pl.Series) (TYPE_CHECKING import, zero ignores inside; isinstance narrows Series.min/max) -- the dispatcher keeps exactly one localized ignore[assignment] at the get_column engine seam (fa9d3f9).
lmeyerov
commented
Aug 3, 2026
| if not isinstance(dtype, np.dtype) or dtype.kind not in "iu": | ||
| # Extension int dtypes (pandas Int64) can hold NA; plain numpy int cannot. | ||
| return None | ||
| null_count = getattr(s_pd, "null_count", None) # cudf: null mask rides a numpy dtype |
Contributor
Author
There was a problem hiding this comment.
The pandas/cudf arm is now _dense_interval_indexable(s: SeriesT): s.dtype is typed directly (no getattr duck-typing); only the cudf-only null_count property keeps a documented getattr, typed as Optional[int] (fa9d3f9).
lmeyerov
commented
Aug 3, 2026
| dtype = getattr(s_pd, "dtype", None) | ||
| if not isinstance(dtype, np.dtype) or dtype.kind not in "iu": | ||
| return False | ||
| null_count = getattr(s_pd, "null_count", None) |
Contributor
Author
There was a problem hiding this comment.
_int_col_bounds_within got the same treatment: typed _bounds_within_polars / _bounds_within_indexable helpers with one localized ignore at the dispatch seam; ignores across both proofs went 5 -> 2, zero casts added, hygiene guard unchanged (fa9d3f9).
Review follow-up (#1844): replace the ignore-sprayed engine arms of _dense_int_domain_interval / _int_col_bounds_within with fully typed per-engine helpers -- _dense_interval_polars / _bounds_within_polars (pl.Series under TYPE_CHECKING, zero ignores inside; isinstance narrows Series.min/max instead of ignore[arg-type]) and _dense_interval_indexable / _bounds_within_indexable (SeriesT for pandas/cudf; s.dtype typed directly, no getattr duck-typing on dtype). Each dispatcher keeps exactly ONE localized ignore[assignment] at the get_column engine seam with a reason comment; the pandas arm's __getitem__ seam needs none. Ignores in these helpers: 5 -> 2; zero casts added (explicit-cast stays at the 133 baseline); type-hygiene guard unchanged; mypy clean modulo pre-existing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
Review follow-up (#1844, test-sufficiency question): pin the dense-domain kernel against an INDEPENDENT pandas merge-join oracle (not its own decline fallback) across: (a) engine-matrix kernel-served vs forced-decline value parity for pandas/polars, plus cudf lanes gated on a cupy.bincount NVRTC probe (skip on cudf-ok/JIT-broken hosts); (b) id-dtype matrix -- int8/int16/ int32/int64/uint32 all serve with identical counts, pandas extension Int64 declines with the reason pinned at both proof seams, float ids decline at both seams; (c) self-loops + duplicate multi-edges multiplicative-counting parity; (d) table-budget boundary exact at n = 4E+1024 (serve at -1 and exactly, decline at +1) and a large-lo/small-width interval through the round-1 shift path (helper and full path, kernel-served proven by spy); (e) seeded differential -- 10 fixed-seed random graphs x {dense-from-0, dense-from-offset, gapped(decline), shuffled-rows} x both engines, verdict recorded via kernel spy and values always equal to the oracle. 31 tests, all deterministic; local: 29 pass + 2 cudf skips; full cypher suite fail-set unchanged vs pre-change baseline (both empty). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
lmeyerov
added a commit
that referenced
this pull request
Aug 3, 2026
Stack sync after the #1844 review follow-ups (typed per-engine dense-proof helpers + T6B review-sufficiency suite). Union resolution: round 2's fused _edge_cols_bounds_within is kept unchanged; the per-column _int_col_bounds_within dispatcher over the typed arms is retained as the seam the T6B tests pin directly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A proof-gated dense-domain kernel for the equal-domain branch of the two-hop
count(*)fast path (graph-benchmark q8 class:MATCH (a {t})-[{r}]->(b {t})-[{r}]->(d {t}) RETURN count(*)). That branch spent its time restricting the rel-filtered edges to domain × domain — two semi-joins on polars, two isin masks on pandas/cuDF — not counting: profiled one-shot on the 20k graph-benchmark (249k FOLLOWS edges), 5.3 ms of the 9.3 ms polars call was that restriction.Why it is safe
When the domain's ids form a dense integer interval
[lo, hi](n_unique == hi - lo + 1) and both endpoint columns are integer, null-free and bounded by[lo, hi], interval membership IS set membership: the restriction is provably the identity, and the degree product collapses to two O(E) bincounts plus one O(N) aligned product-sum. The proof is a handful of O(E) min/max reductions — a data property, not a query-shaped hack (dense type-partitioned ids are the idiomatic multi-table graph encoding). Any failed guard declines to the existing memoized semi-join path unchanged: out-of-domain endpoints, gapped/non-integer/null ids, empty edges, and a memory guard for domains far wider than the edge count. Engine-polymorphic via the index module's array helpers (numpy host for pandas/polars/polars-gpu, cupy for cudf). When the kernel serves, no cross-call memo is written, so one-shot and warm calls converge — removing the #1825 binding sensitivity for this shape.Tests: T6 lane pin (kernel serves + memo path provably not entered), value parity vs the forced-decline semi-join path, offset and negative interval shifts, every decline guard pinned, and the memo-lane reachability pin under the new seam.
Measured cells (attributed: q8, both scales)
Measurements come from the COMBINED candidate build
938f22851e68918799da2bee49c88db2e2c103ee(masterf875724ce+ this branch +perf/gfql-singlehop-subsumed-semijoin+perf/gfql-q7-two-star-minimal-join+perf/gfql-q8-bincount-r2), with per-fix attribution by disjoint cell sets: this PR owns q8 (jointly with the stacked round-2 branch at 100k), the single-hop PR owns q1/q3/q4, the two-star PR owns q5/q6/q7.Comparator quotes, baseline boards (
results/graphbench-board-{20k,100k-v2}-20260802) vs candidate lanes (results/graphbench-board-{20k,100k}-cand-20260803, graphistry/pyg-bench#170):LOSE 2.94xWIN 1.27x (WEAK: slot ranges overlap)LOSE 2.44xLOSE 1.52x(narrowed; includes the stacked shift-elision round 2)Untouched cells did not move: cells outside this PR's attribution set changed only where attributed to the sibling branches above; the two unattributed cells (q2, q9) kept their WIN verdicts at both scales (q9 within noise; q2's improvement rides the same fused single-hop lane as the semi-join subsumption branch).
Receipts
results/graphbench-board-20k-cand-20260803/andresults/graphbench-board-100k-cand-20260803/(compare.txt, SOURCE_COMMITS, lock-receipt, 1 Hz load receipts, SHA256SUMS).938f22851e68918799da2bee49c88db2e2c103ee, pyg-benchfae1e975.Do not merge without owner review.
🤖 Generated with Claude Code
https://claude.ai/code/session_011AB4RZpph3uSFUpzKnZJcr