GFQL: #1658 index does not engage on polars/polars-gpu direct hops (frame-identity churn)

[lmeyerov]

Summary

The #1658 CSR adjacency index accelerates seeded hop() on pandas and cudf but never engages on polars / polars-gpu for direct hops: every hop falls back to the O(E) scan even when a resident, valid index was just built via gfql_index_all(engine="polars"). Correctness/parity is unaffected (the scan result is identical) — this is purely a lost fast path. cudf is affected only for a narrow cost-gate case (below); polars/polars-gpu miss on every covered hop.

Evidence (index_trace, qualitative)

Wrapping identical single-seed single-hop queries in index_trace():

• pandas: every hop records path="index" (the feat(gfql): physical adjacency indexes for O(degree) seeded traversal #1658 CSR engages).
• cudf: mostly path="index"; one low-cardinality single-hop query records path="scan" with reason "frontier N >= 0.02*n_keys (...) -> scan cheaper".
• polars / polars-gpu: all hops record path="scan", used_index=False, and the trace shows ~2 steps per hop (vs 1 on pandas).

The doubling is diagnostic: maybe_index_hop runs at two sites — the shared hook in compute/hop.py and again in lazy/engine/polars/hop.py (so chain_polars gets the hook too). On a hit the first call returns early (1 step); on a bail both fire (2 steps). So 2 steps == a confirmed bail.

Root cause (polars/polars-gpu): edge-frame identity churn

registry.get_valid() validates a resident index primarily by frame identity (idx.source_ref is df). hop() runs _coerce_input_formats(self, engine) before the index hook. For native-polars frames that helper unconditionally re-wraps the edges/nodes through _pl_nan_to_null, and _pl_nan_to_null returns a fresh frame object whenever the frame has any float column (it keys on column presence, not on whether a NaN is actually present).

Because gfql_index_all builds three indexes (each coercing) and .hop() coerces once more, the live edges frame is a different object on every step. So each adjacency index's source_ref no longer matches the live edges frame → get_valid returns None → _indices_for_direction returns None → index_seeded_hop returns None → scan. pandas and cudf don't hit this: their coercion returns the same frame object (pandas identity / cudf→cudf identity); only the polars tail rebuilds.

Confirmed by a minimal synthetic graph: with a (NaN-free) float edge column, polars records all-scan and source_ref is edges is already False at build time; without the float column, polars records path="index". Making _pl_nan_to_null rebuild only when a NaN is actually present flips the float-column case back to path="index".

Root cause (cudf, the one scan)

cudf keeps frame identity, so its indexes stay valid — the single scan is the cost gate: the index-vs-scan crossover fraction is 0.5 for pandas but 0.02 for cudf and polars. For a single-node seed (frontier=1) the gate trips when the keyed endpoint has <= ~50 distinct values, i.e. the lowest-cardinality single-edge-type slice. Cost-model artifact, not a residency miss.

Scope / design

• Primary (small, low-risk): make _pl_nan_to_null identity-stable — rebuild only float columns that actually contain NaN, return the input unchanged otherwise. Preserves the pandas-oracle NaN->null parity semantics while removing the needless rebuild that churns frame identity. Benefits all polars paths, not just the index. Verified to flip polars from all-scan to index.
• Secondary (small): revisit the 0.02 polars/cudf cost-gate fraction so tiny typed subgraphs still index single-seed queries (n_keys-aware floor).
• Cosmetic (small): de-duplicate the two maybe_index_hop call sites so a bail isn't retried and the trace isn't doubled.
• Deferred (large): device-resident CSR for polars-gpu (today host numpy CSR + host gather; fine functionally, a perf refinement only).

The engine-polymorphic array layer already supports polars (numpy host arrays, get_column().to_numpy(), positional df[idx] gather, semi-join select), so no new polars-resident index representation is required for the fast path to engage.

Relation to other work

• GFQL/polars: native rows(binding_ops) — traversal queries NIE (blocks polars for edge/multi-hop Cypher) #1709 (polars native binding-ops): the second maybe_index_hop site lives on the polars hop entry that GFQL/polars: native rows(binding_ops) — traversal queries NIE (blocks polars for edge/multi-hop Cypher) #1709 builds out; that path can only benefit from the index once this identity churn is fixed.
• GFQL: route Cypher chain node-id filter through resident node_id index (O(1) point / O(log) range seek) #1676 (route chain/Cypher through the index): routing chain/Cypher through the index hook is moot on polars until the index actually engages — this is a prerequisite for GFQL: route Cypher chain node-id filter through resident node_id index (O(1) point / O(log) range seek) #1676 delivering on polars/polars-gpu.

Key file:line anchors (pinned tree)

• Bail path: compute/hop.py:110,117-136 · compute/gfql/index/api.py:342-473 (cost gate :443-452, index_seeded_hop call :459) · compute/gfql/index/traverse.py:42-44,74-76 · compute/gfql/index/registry.py:111-126 (identity guard :122).
• Root cause (churn): compute/ComputeMixin.py:108-113 (unconditional re-wrap) · Engine.py:210-222 (_pl_nan_to_null, float-presence check :219, fresh-object :222); pandas/cudf identity Engine.py:236-237,268-269.
• Double invocation: compute/gfql/lazy/engine/polars/hop.py:20-43 (2nd call :26).
• Cost gate: compute/gfql/index/cost.py:16-17,63-75.