Cypher/GFQL: introduce a first-class row-carrier IR for multi-stage vectorized row semantics

[lmeyerov]

Problem

The current local Cypher/GFQL execution model does not have a first-class row-carrier IR.

As a result, row-seeded semantics are handled in feature-specific ways instead of through one reusable contract. The bounded reentry work in PR #975 is the clearest example, but the same architectural pressure will recur for:

• multi-stage MATCH ... WITH ... MATCH ...
• multi-alias WITH / RETURN / ORDER BY
• OPTIONAL MATCH null extension
• grouped row-preserving aggregation
• future vectorized GPU/backend audits

Today those semantics are spread across compiler rewrites, projection metadata, hidden columns, and runtime stitching. That is workable for a bounded slice, but it is not a good long-term model for a graph query compiler/runtime.

Why This Matters

• makes row semantics harder to extend cleanly
• encourages feature-by-feature protocol glue instead of a reusable IR
• complicates vectorization reasoning and GPU/backend parity review
• increases the chance that future Cypher support growth expands lowering.py and gfql_unified.py in ad hoc ways

Proposed Direction

Introduce a first-class row-carrier / seeded-row IR for local Cypher/GFQL execution.

Core ideas:

1. Represent carried row state explicitly.

   • row ids / seed ids
   • bound aliases
   • carried scalar columns
   • ordering contract
   • null-extension contract where applicable

2. Lower row-seeded features into that IR instead of feature-specific side channels.

3. Keep the implementation columnar/vectorized.

   • pandas/cudf-friendly
   • no generic Python row-loop fallback

4. Let specific features become clients of the same row model.

   • bounded reentry
   • later multi-alias WITH
   • later OPTIONAL MATCH
   • later multiplicity-preserving grouped aggregation

Relationship To Other Issues

• Cypher/GFQL: replace bounded reentry hidden-column handshake with an explicit ReentryPlan #987 is the smaller, immediate cleanup issue for the current bounded-reentry implementation.
• This issue is the broader architecture lane that goes beyond bounded reentry and aims at a reusable row-semantics model.
• The two should inform each other, but this issue is intentionally larger in scope.

Non-Goals

• not a request to reopen PR fix: extend bounded cypher reentry carrier #975
• not a demand for immediate semantic expansion
• not a license to add non-vectorized fallback execution

Success Criteria

• bounded reentry can be expressed as a normal client of the row IR
• future row-seeded Cypher features stop requiring bespoke hidden-column / metadata handshakes
• vectorization/backend expectations are clearer to audit
• lowering.py and runtime orchestration can shrink over time instead of accumulating one-off row mechanics

Context

PR #975 is landing the bounded-reentry feature/hardening slice.
Issue #987 tracks the narrower follow-on cleanup for that implementation.
This issue tracks the broader architectural direction beyond that one feature.

[lmeyerov]

Related narrower follow-on: #987 tracks the bounded-reentry-specific cleanup that can land independently of the broader row-carrier IR direction in this issue.

[lmeyerov]

Concrete row-carrier invariants blocking LDBC SNB IC3 / IC5

Cross-link from #999. While reproducing the multi-stage MATCH ... WITH ... MATCH ... re-entry gaps on master@6891d39e4, the reduced repros did not stop at the documented parser surface gates — they fell through to two row-lowering invariants that sit at the heart of the row-carrier IR design this issue is scoped to:

1. graphistry/compute/gfql/cypher/lowering.py:7622 — "Cypher MATCH after WITH currently requires the trailing MATCH to start from the same carried node alias." Today's bounded-reentry contract assumes every re-entry MATCH starts from the carried alias. LDBC IC3 explicitly carries (a, x, y) across an intermediate MATCH (city:City)-[:IS_PART_OF]->(country:Country) whose source alias city is not carried — the row carrier needs to forward unrelated bound aliases through stages where they aren't the MATCH source.

2. graphistry/compute/gfql/cypher/lowering.py:1914 (via _validate_row_expr_scope → _validate_aggregate_expr_scope → _lower_general_row_projection) — "Cypher row lowering currently supports one MATCH source alias at a time." RETURNing from two distinct MATCH source aliases in the same projection is rejected. LDBC IC5 hits this directly: RETURN forum.id, post.id after MATCH chains rooted at different sources.

These are not parser surface concerns — they are the exact "feature-specific protocol glue" this issue calls out. If ReentryPlan (#987) and the broader row-carrier IR (this issue) get an explicit multi-alias carry contract, both invariants relax naturally and the LDBC SNB IC3/IC5/IC10 family unlocks together.

#999 is being scoped narrowly to the parser-surface lift (Path A in that thread) so it doesn't drift into a row-carrier rewrite. The architectural exit gate for the SNB multi-stage benchmark family is here.

[lmeyerov]

Picking this up — Slice 1 scope

Following up on the earlier comment about LDBC IC3/IC5 invariants landing here: starting active work on this issue under branch feat/989-reentryplan-multi-alias-carry (cut from master@6891d39e4). Plan file under plans/989-reentryplan-multi-alias-carry/.

Slice 1 scope (this PR)

• Introduce explicit ReentryPlan dataclass per #987 Step 1 — replaces the tuple returns and side-channel metadata that the bounded-reentry path threads today.
• Replace the hidden-column handshake (__cypher_reentry_*) with an explicit carried-column contract on the plan (#987 Step 2).
• Move bounded-reentry runtime stitching out of gfql_unified.py into a focused module (#987 Step 3).
• Extend the carrier to N whole-row aliases — the IC3 unblock from #999. When prefix WITH projects multiple whole rows (e.g. WITH a, x, y), one becomes the trailing-MATCH source and the rest are carried as scalar bundles addressable downstream as x.id, x.name, etc.
• IC3-shaped pandas + cuDF tests against an LDBC-mini Person/Country/City fixture.
• Failfasts for shapes still out of scope.

Out of scope (Slice 2, follow-on PR)

• Multi-MATCH-source row projection (lowering.py:1914) — the IC5 unblock.
• Variable-length relationship reentry under multi-alias carry (overlaps #1009).
• Generic WITH semantic broadening — preserving the non-goal called out in this issue.

Coordination note

If anyone is touching lowering.py _compile_bounded_reentry_query / _bounded_reentry_carry_columns or gfql_unified.py reentry stitching, please ping — that's the surface area being rewritten. Other parts of lowering.py (general row projection, optional match, varlen) are untouched.