perf(gfql): secondary node property indexes for seeded traversal

[lmeyerov]

Stacked on #1776 (base branch perf/gfql-general-indexed-bindings) — review that one first.

Why

With #1776 landed, official LDBC SNB SF1 interactive-short IS7 was 72 ms against Kuzu 36.7 / Ladybug 36.0. Phase timing (plain perf_counter, not cProfile) located the whole gap precisely:

phase	calls	ms
indexed helper	1	57.2
— of which _filter_frame	5	51.5
everything else (index lookups, gathers, orientation, joins, estimates)	—	< 2

and per-call shapes showed a single offender:

filter	input	output	ms
seed {id, label__Message}	3,181,724 × 32	1	51.8
edge {type: REPLY_OF}	20 × 8	19	0.27
node {label__Comment}	19 × 32	19	0.21
edge {type: HAS_CREATOR}	57 × 8	19	0.22
node {label__Person}	15 × 32	15	0.19

The traversal is already optimal. We were scanning 3.2M rows to find one seed node, because the graph's node id is bound to a synthetic key while the query seeds on the id property — and the registry had no index for properties. That's the secondary-index capability other engines have; this PR adds it.

What

• NodePropIndex — sorted distinct values over node row positions, CSR form so duplicate values are indexable (unlike the unique-only node-id index). Never reorders .nodes; identity + fingerprint validated, so a .nodes() rebind is a safe miss; engine-polymorphic (numpy host / cupy on-device). Kept in a column-keyed map beside the kind-keyed one.
• build_node_prop_index declines (→ scan) for anything whose vectorized ordering/equality isn't unambiguous on both backends — today integer dtypes only (float NaN ordering, cupy strings, nulls). Widening is additive.
• lookup_prop_rows (searchsorted + CSR range expansion) and prop_match_count, a free selectivity estimate read straight off the CSR offsets.
• Planner: the seeded fixed-hop path picks the most selective indexed scalar predicate in the seed filter, gathers those candidates, and hands them to the unchanged filter, which applies every remaining predicate. Identical results whether the index is present, absent, stale, or cost-gated out.
• Surface: create_index('node_prop', column=...) (column required), drop_index('node_prop', column=...), show_indexes() rows, g.gfql_index_node_props([...]) (skips unindexable columns).

Opt-in, exactly like other engines' secondary indexes: with no property index resident, behaviour and cost are unchanged.

Perf (dgx-spark, official LDBC SNB SF1, warm medians, value-identical 19 rows)

	before	after
IS7 end-to-end	71.6 ms	19.5 ms (3.7×)
seed lookup alone	51.2 ms	0.096 ms
one-time index build	—	112 ms

That puts IS7 ~1.9× ahead of the frozen Kuzu (36.65 ms) and Ladybug (36.0 ms) reference points, from ~2× behind.

A position-balanced A/B of IS1–IS7 with no property index resident shows no detectable change — per-query deltas sit inside each build's own 6–10% run-to-run spread, and with nothing indexed the added path is an empty-tuple check.

Tests

Standard-derived, all against the same-engine canonical path: seed-without-scanning parity plus candidate width on pandas/polars/cuDF; indexed-vs-absent equality; duplicate-value keys gathering every matching row; stale-rebind and policy-off fallback; unindexable dtype / missing column / missing argument errors alongside the skipping convenience wrapper; most-selective-column choice; show/drop lifecycle.

Validation (dgx-spark, capped guard, read-only source)

• full CPU graphistry/tests/compute: 6242 passed, 1 failed — master's pre-existing test_chain_dask_edges
• cuDF lane: 62 passed
• ruff: clean · mypy: 0 new errors vs master

🤖 Generated with Claude Code

https://claude.ai/code/session_01NxxTFpKiqA4FQa9Pjy8yR1

---

Review follow-ups (2026-07-25)

Manual review of the stack raised DRY/placement, functional-style, dynamic-typing, and test-amplification concerns. Addressed in 189dcc73 (this PR) and 83212ab1 (on #1776):

DRY / placement — bindings._concat duplicated Engine.df_concat (whose polars twin already accepts ignore_index for exactly this) and is deleted; _estimate_join_rows / _join_state / _filter_oriented_endpoints move to compute/dataframe/join.py as estimate_inner_join_rows / path_ordered_expand_join / semijoin_by_column, generalized to explicit column parameters; _lookup_degree moves to index/lookup.py and now shares one CSR probe with the property lookup (three copies of searchsorted+membership became _csr_hit_positions / csr_match_count / csr_gather_rows). bindings.py drops from 17 local helpers to 12, all index-specific.

Functional style — copy-then-mutate became a single batched .assign(...) throughout, plus a redundant .copy() dropped before .rename() and after a boolean mask. Rule written down in the review skill.

Dynamic typing — typed handoff (see #1776), _integer_index narrows with isinstance on the index union instead of probing getattr(..., "keys_sorted", None), alias/filter access narrows on ASTNode/ASTEdge. Across the stack: getattr/setattr 31 → 19 (5 of those inside the one module that owns the attribute; the rest are threading.local probes and pre-existing _gfql_rows_* plumbing), # type: ignore 18 → 9, and mypy --warn-unused-ignores reports none of the remaining ones unnecessary.

A correctness fix the new tests caught — _connected_decline_reason re-derived the cost gate locally, so on engines with a tighter crossover (0.02 vs pandas' 0.5) a structurally unsupported shape was reported as cost_frontier. It now asks the real gate with cost disabled: a serve there means cost was the only obstacle, anything else is structural. One source of truth, accurate on every engine.

Tests — the unsupported-shape decline matrix now runs pandas/polars/cuDF (was pandas-only; that is what caught the mislabelling); a negative matrix for the polars early gate (rows-with-source, alias-endpoints, alias-prefilters, seeded re-entry, plan mismatch, unnamed middle) plus a positive assertion so it cannot go vacuous; the property-index cost gate is now exercised under use with the crossover pinned via the public knob (every other case used force, which skips the gate); and the property index is now reachable from all three surfaces — the Cypher DDL grammar accepts node_prop, and the JSON wire protocol handles column-keyed create/drop/name resolution (it previously raised "no resident index" for a resident property index and dropped every column on a targeted drop). docs/source/gfql/indexing.rst documents the new kind.

Verified: full CPU graphistry/tests/compute 6260 passed with only master's pre-existing test_chain_dask_edges; cuDF lane 70 passed; ruff clean; mypy unchanged vs master.

Not in this PR: benchmark scaffolding stays out of this repo — the reusable IS1–IS7 A/B suite config and the phase-timing probe went to pyg-bench (bench/gfql-is1-is7-ab-harness). Engine-shape divergences found along the way are filed as #1779.

Review follow-ups, round 2

Two further nits, both about hiding things the type system should carry:

• Declared fields (7777e191, on perf(gfql): dispatch indexed fixed-hop bindings before the canonical traversal #1776): with_index_policy no longer does setattr(out, POLICY_ATTR, policy) — the whole _gfql_* execution context is declared on Plottable/PlotterBase, so index/handoff.py has zero dynamic attribute calls and the policy/registry accessors are plain typed access. Two latent bugs surfaced doing it (a Plottable stand-in missing fields; a getattr fallback silently disabled by the new default) — details in perf(gfql): dispatch indexed fixed-hop bindings before the canonical traversal #1776.
• Narrowed suppression (d85b421f, here): gfql_index_node_props wrapped create_index in except ValueError, swallowing caller mistakes — a typo'd column name silently left the query unindexed. create_index now raises the purpose-built GfqlIndexUnsupportedError (a ValueError subclass, so existing handlers keep working) for exactly the two conditions the DATA cannot support: duplicate node ids and an unindexable property dtype. The convenience builders catch only that; everything else propagates. gfql_index_all gets the same treatment for node_id. Tests pin both directions — unindexable dtypes are skipped, a missing column raises, and the raised type is asserted to not be the skippable one.

Full-stack verification from a pristine master clone: CPU 6260 passed (only master's pre-existing test_chain_dask_edges), cuDF 70 passed, ruff clean, mypy unchanged vs master, and mypy --warn-unused-ignores reports no unnecessary suppression from this stack (the 7 it does flag exist verbatim on master).

Review follow-ups, round 3

Three more nits, all on files owned by #1776 and fixed there (54d6e77f, 6c41ee8e), then merged forward:

• row/frame_ops.py's RowPipelineCtx protocol declared its fields Any → now DataFrameT / str / Plottable / Iterable[str] via a TYPE_CHECKING import, and the file has zero getattr.
• polars/row_pipeline.py swallowed unexpected exceptions — the extracted finisher made the indexed path share the generic path's except SchemaError: return None. It now takes decline_on_schema_error, so only the generic caller may decline; on indexed state (whose dtypes our helper already verified) a SchemaError raises.
• The execution-context field conversion is complete: zero non-test dynamic access to the family repo-wide, and both hand-rolled Plottable stand-ins now carry _gfql_index_registry, which get_registry reads directly.

Full-stack verification from a pristine master clone: CPU 6260 passed (only master's pre-existing test_chain_dask_edges), cuDF 70 passed, ruff clean, mypy unchanged vs master.

[lmeyerov]

revisit for static typing..

[lmeyerov]

avoid or make more managed