Skip to content

Fix silent row loss when LOAD FROM/UNWIND feeds a MATCH primary key predicate - #864

Merged
adsharma merged 2 commits into
mainfrom
fix_load_from
Aug 30, 2026
Merged

Fix silent row loss when LOAD FROM/UNWIND feeds a MATCH primary key predicate#864
adsharma merged 2 commits into
mainfrom
fix_load_from

Conversation

@adsharma

Copy link
Copy Markdown
Contributor

Summary

Fixes #861

LOAD FROM <source> followed directly by MATCH (n:Label {pk: col}) — where col is a scan column — matched zero rows silently on 0.19.0–0.20.0. No error was raised, the query reported success, and any SET/MERGE hanging off the MATCH never executed:

LOAD FROM 'updates.csv' (header=true)
MATCH (p:Products {handle: handle})
SET p.description = description
RETURN p.handle   -- 0 rows; p.description unchanged

Root cause

The planner legitimately rewrites a single-node MATCH with a primary-key equality predicate into a QUERY_PRIMARY_KEY_LOOKUP whose key is evaluated on top of the outer (scan) pipeline. However, ProjectionPushDownOptimizer never visited LogicalQueryPrimaryKeyLookup, so its key expression was never collected as in-use. visitTableFunctionCall then marked the referenced scan column as skippable (columnSkips) and the physical scan stopped writing that vector. The key evaluator subsequently read an unwritten vector, the key evaluated to NULL for every row, lookupPK failed, and the lookup returned an empty result — silently.

This also explains the report's observations:

  • LOAD FROM df MATCH (p:Products {handle: handle}) SET ... failed silently (the key column was pruned from the scan).
  • An explicit WITH handle AS h between the scan and the MATCH worked: the intermediate PROJECTION starts a fresh push-down pass that keeps the column alive.
  • Matching on a non-primary-key property ({description: description}) worked: it becomes a regular filter, which collects its predicate.
  • MATCH (p:Products {handle: 'a'}) (literal key) worked: no outer dependency, no pruning.

Fix

Override visitQueryPrimaryKeyLookup in ProjectionPushDownOptimizer to collect the lookup's key and node-ID expressions as in-use, so the scan column they reference is kept alive. This works for both the raw-variable case (the issue's {pk: col}) and arbitrary key expressions (e.g. the implicit CAST inserted for a SERIAL primary key), since collectExpressionsInUse recurses into children.

Testing

  • Added regression cases to test/test_files/load_from/load_from.test:
    • LOAD FROM ... MATCH (p:person {ID: id}) RETURN COUNT(*) → 5 (property-map form)
    • LOAD FROM ... MATCH (p:person) WHERE p.ID = id RETURN COUNT(*) → 5 (equivalent WHERE form)
    • LOAD FROM ... MATCH (p:person {ID: id}) SET p.fName = fName RETURN p.fName ORDER BY id → 5 rows (verifies SET executes through the lookup)
  • e2e_test full run: 1960 passed; the only 2 failures (copy_to_csv.CopyToInvalidCase, dictionary_bug~...AnonymousParquetDeleteReload) are pre-existing environment issues (missing dataset/empty fixtures; JSON export extension not built) and fail identically without this change.
  • optimizer_test, planner_tests, binder_test, api_test all pass.

…redicate

Fixes #861

The projection push-down optimizer did not visit the key expression of
LogicalQueryPrimaryKeyLookup, so a scan column referenced only as the
lookup key was marked as unused and pruned from the scan's columnSkips
(e.g. LOAD FROM csv/table-function output). The scan then left the key
vector unwritten, the key evaluated to NULL for every row, and the
lookup silently returned zero rows - dropping any MATCH/SET/MERGE
hanging off it without error.

Query shapes affected (all silently matched 0 rows):

    LOAD FROM 'updates.csv' (header=true)
    MATCH (p:Products {handle: handle})
    SET p.description = description

    UNWIND [...] AS x MATCH (p:Products {handle: x}) ...

The optimizer now collects the lookup's key (and node ID) expressions
as in-use, keeping the referenced scan column alive. Explicitly visit
QueryPrimaryKeyLookup in ProjectionPushDownOptimizer.

Adds regression cases to load_from.test covering the property-map
predicate, the equivalent WHERE form, and a SET through the lookup.
While auditing ProjectionPushDownOperator coverage after the primary key
lookup fix, two more operators were found that consume expressions which
are not part of their own output schema but were silently skipped by the
visitor (they were not even present in LogicalOperatorVisitor's dispatch
switch):

- INDEX_LOOK_UP (LogicalPrimaryKeyLookup): evaluates key expressions on
  top of the copy-from source scan. Currently safe only because
  visitCopyFrom happens to collect all source columns and COPY_FROM
  always sits above it; any change to key binding would reintroduce the
  same silent-NULL failure mode as #861.

- UNWIND_DEDUPLICATE: consumes keyExpressions that are not part of its
  output schema; currently kept alive only because they overlap with
  expressions collected through the MERGE above it.

Add dispatch cases for both operators to LogicalOperatorVisitor and
override them in ProjectionPushDownOptimizer to collect the lookup keys
(and warning expressions) and the dedup keys as in-use.
@adsharma
adsharma merged commit 69b3290 into main Aug 30, 2026
4 checks passed
@adsharma
adsharma deleted the fix_load_from branch August 30, 2026 17:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: LOAD FROM does not bind into a following MATCH property predicate — writes silently lost

1 participant