Skip to content

test: cover ambiguous exact nested Parquet field matches - #5751

Open
peterxcli wants to merge 1 commit into
apache:mainfrom
peterxcli:test/parquet-ambiguous-exact-field-match
Open

test: cover ambiguous exact nested Parquet field matches#5751
peterxcli wants to merge 1 commit into
apache:mainfrom
peterxcli:test/parquet-ambiguous-exact-field-match

Conversation

@peterxcli

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #5707.

Rationale for this change

Comet can retain DataFusion's generic CastExpr for pure structural narrowing of nested Parquet columns, allowing DataFusion's nested leaf pruning to read only the requested fields. Retaining that cast is safe only when DataFusion's exact-name lookup agrees with Spark's configured field-name resolver.

An exact match alone is insufficient: with case-insensitive resolution, a file containing s: struct<ID: bigint, id: bigint> and a requested schema of s: struct<id: bigint> is ambiguous. DataFusion's generic cast can select the exact id field, whereas Spark and Comet's Parquet converter reject the duplicate match. The same problem applies to non-ASCII names such as CAFÉ and café, and to structs nested inside other structs or lists.

The implementation merged in #5262 already requires exactly one source match under Spark's configured resolver at each nested struct level, in addition to an exact-name match. Ambiguous narrowing therefore falls back to CometCastColumnExpr, preserving the existing duplicate-field error. This PR adds the missing regression coverage for that behavior; it does not change production code.

What changes are included in this PR?

  • Add a native regression covering ASCII and non-ASCII sibling names with an exact requested match, directly in a struct and inside nested structs and lists.
  • Check both the narrowing predicate and the actual expression-adapter rewrite. Case-insensitive ambiguity must select CometCastColumnExpr; case-sensitive exact matches must retain DataFusion's CastExpr so valid pruning remains enabled.
  • Parameterize the existing native-reader regression to request both mixed-case Café and exact-case café from CAFÉ/café siblings. Both reads must raise a duplicate-field error, and the test asserts that the plan uses CometNativeScanExec.

How are these changes tested?

  • 14 native structural-narrowing tests passed.
  • 2 focused CometNativeReaderSuite tests passed on Spark 4.1.
  • Rust formatting, Maven Spotless, and git diff --check passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nested Parquet cast retention accepts an ambiguous case-insensitive field match

1 participant