[SPARK-57725][SQL][4.1] Fix NPE in AttributeSeq column resolution when an attribute has a null name#56838
Closed
MaxGekk wants to merge 1 commit into
Closed
[SPARK-57725][SQL][4.1] Fix NPE in AttributeSeq column resolution when an attribute has a null name#56838MaxGekk wants to merge 1 commit into
MaxGekk wants to merge 1 commit into
Conversation
…attribute has a null name
### What changes were proposed in this pull request?
`AttributeSeq` builds case-insensitive name lookup maps (`direct`, `qualified`, `qualified3Part`, `qualified4Part`) by grouping the attributes on `attr.name.toLowerCase(Locale.ROOT)`. The grouping key function dereferences the name without a null check, so a single attribute whose name is `null` makes `groupBy(_.name.toLowerCase(...))` throw a `NullPointerException`, aborting resolution of the whole operator with an `INTERNAL_ERROR` (SQLSTATE XX000) instead of resolving the other columns.
This PR introduces a `namedAttrs` lazy val that filters out null-named attributes, and builds the four name maps from it instead of from `attrs`. Positional and expression-id access (`apply(ordinal)`, `indexOf(exprId)`) still use the full `attrs`, so they are unaffected.
### Why are the changes needed?
A null-named attribute can arise on the JVM side: `StructField` permits a null name (no `require(name != null)`), and the name flows unchanged through `DataTypeUtils.toAttribute` into `AttributeReference`. For example:
```scala
val schema = StructType(Seq(StructField(null, IntegerType), StructField("b", IntegerType)))
val df = spark.createDataFrame(spark.sparkContext.parallelize(Seq(Row(1, 2))), schema)
df.select("b").collect() // forces resolution -> NPE / INTERNAL_ERROR
```
A null-named attribute is unaddressable by any column reference (a reference's name parts are never null), so dropping it from the name maps cannot change resolution of any legitimate reference. It converts the hard `NullPointerException` into correct resolution of the remaining (named) attributes, or a normal unresolved-column error if the null-named column is referenced.
### Does this PR introduce _any_ user-facing change?
No. It only turns an internal `NullPointerException` / `INTERNAL_ERROR` into normal column-resolution behavior.
### How was this patch tested?
Added a regression test `SPARK-57725: resolution does not fail when an attribute has a null name` in `AttributeResolutionSuite`, exercising the `direct`, `qualified`, `qualified3Part`, and `qualified4Part` maps with a null-named sibling attribute.
```
build/sbt 'catalyst/testOnly *AttributeResolutionSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
Closes apache#56831 from MaxGekk/fix-npe.
Authored-by: Maxim Gekk <max.gekk@gmail.com>
Signed-off-by: Max Gekk <max.gekk@gmail.com>
(cherry picked from commit 13b3237)
Signed-off-by: Max Gekk <max.gekk@gmail.com>
(cherry picked from commit 6754a42)
dongjoon-hyun
approved these changes
Jun 27, 2026
Member
Author
|
Merging to 4.1. Thank you, @dongjoon-hyun for review. |
MaxGekk
added a commit
that referenced
this pull request
Jun 28, 2026
…n an attribute has a null name ### What changes were proposed in this pull request? This is a backport of #56831 to `branch-4.1`. `AttributeSeq` builds case-insensitive name lookup maps (`direct`, `qualified`, `qualified3Part`, `qualified4Part`) by grouping the attributes on `attr.name.toLowerCase(Locale.ROOT)`. The grouping key function dereferences the name without a null check, so a single attribute whose name is `null` makes `groupBy(_.name.toLowerCase(...))` throw a `NullPointerException`, aborting resolution of the whole operator with an `INTERNAL_ERROR` (SQLSTATE XX000) instead of resolving the other columns. This PR introduces a `namedAttrs` lazy val that filters out null-named attributes, and builds the four name maps from it instead of from `attrs`. Positional and expression-id access (`apply(ordinal)`, `indexOf(exprId)`) still use the full `attrs`, so they are unaffected. ### Why are the changes needed? A null-named attribute can arise on the JVM side: `StructField` permits a null name (no `require(name != null)`), and the name flows unchanged through `DataTypeUtils.toAttribute` into `AttributeReference`. A null-named attribute is unaddressable by any column reference (a reference's name parts are never null), so dropping it from the name maps cannot change resolution of any legitimate reference. It converts the hard `NullPointerException` into correct resolution of the remaining (named) attributes, or a normal unresolved-column error if the null-named column is referenced. ### Does this PR introduce _any_ user-facing change? No. It only turns an internal `NullPointerException` / `INTERNAL_ERROR` into normal column-resolution behavior. ### How was this patch tested? Backported regression tests in `AttributeResolutionSuite` (covering the `direct`, `qualified`, `qualified3Part`, and `qualified4Part` maps) and an end-to-end test in `DataFrameSuite`. The same tests pass on the `branch-4.2` backport (#56837); CI runs them here. ``` build/sbt 'catalyst/testOnly *AttributeResolutionSuite' build/sbt 'sql/testOnly *DataFrameSuite -- -z "SPARK-57725: resolve columns when the input plan has a null-named attribute"' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Closes #56838 from MaxGekk/backport-SPARK-57725-4.1. Authored-by: Maxim Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is a backport of #56831 to
branch-4.1.AttributeSeqbuilds case-insensitive name lookup maps (direct,qualified,qualified3Part,qualified4Part) by grouping the attributes onattr.name.toLowerCase(Locale.ROOT). The grouping key function dereferences the name without a null check, so a single attribute whose name isnullmakesgroupBy(_.name.toLowerCase(...))throw aNullPointerException, aborting resolution of the whole operator with anINTERNAL_ERROR(SQLSTATE XX000) instead of resolving the other columns.This PR introduces a
namedAttrslazy val that filters out null-named attributes, and builds the four name maps from it instead of fromattrs. Positional and expression-id access (apply(ordinal),indexOf(exprId)) still use the fullattrs, so they are unaffected.Why are the changes needed?
A null-named attribute can arise on the JVM side:
StructFieldpermits a null name (norequire(name != null)), and the name flows unchanged throughDataTypeUtils.toAttributeintoAttributeReference. A null-named attribute is unaddressable by any column reference (a reference's name parts are never null), so dropping it from the name maps cannot change resolution of any legitimate reference. It converts the hardNullPointerExceptioninto correct resolution of the remaining (named) attributes, or a normal unresolved-column error if the null-named column is referenced.Does this PR introduce any user-facing change?
No. It only turns an internal
NullPointerException/INTERNAL_ERRORinto normal column-resolution behavior.How was this patch tested?
Backported regression tests in
AttributeResolutionSuite(covering thedirect,qualified,qualified3Part, andqualified4Partmaps) and an end-to-end test inDataFrameSuite. The same tests pass on thebranch-4.2backport (#56837); CI runs them here.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor