[SPARK-39417][SQL] Handle Null partition values in PartitioningUtils #36810
Conversation
singhpk234
force-pushed
the
psinghvk/fix-npe
branch
from
fabe42f
to
abe873f
Compare
singhpk234
force-pushed
the
psinghvk/fix-npe
branch
from
abe873f
to
beba90e
Compare
singhpk234
changed the title
singhpk234
changed the title
|
LGTM. Pending test results |
|
The Python doc generation failure seems to be irrelevant. All the other tests passed. |
|
do we know which commit caused this issue? is it a 3.3 only bug? |
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala
Outdated
Show resolved
Hide resolved
@cloud-fan, This seems to be introduced via commit, SPARK-35561 and seems to only affect |
...cala/org/apache/spark/sql/execution/datasources/parquet/ParquetPartitionDiscoverySuite.scala
Outdated
Show resolved
Hide resolved
|
The python documentation failure is not related to this PR. All the other tests run successfully. I will merge this PR. |
### What changes were proposed in this pull request? We should not try casting everything returned by `removeLeadingZerosFromNumberTypePartition` to string, as it returns null value for the cases when partition has null value and is already replaced by `DEFAULT_PARTITION_NAME` ### Why are the changes needed? for null partitions where `removeLeadingZerosFromNumberTypePartition` is called it would throw a NPE and hence the query would fail. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added a UT, which would fail with an NPE otherwise. Closes #36810 from singhpk234/psinghvk/fix-npe. Authored-by: Prashant Singh <psinghvk@amazon.com> Signed-off-by: huaxingao <huaxin_gao@apple.com> (cherry picked from commit dcfd9f0) Signed-off-by: huaxingao <huaxin_gao@apple.com>
|
Merged to master/3.3. Thanks a lot @singhpk234 and et al. |
|
@singhpk234 is this your first Spark contribution? I tried to assign the jira to you but somehow can't find you. Then I tried to add you to the Spark contributors list but found multiple users with the same name. Can you have your email? |
yup
sure, my email is prashant010696@gmail.com Many thanks @huaxingao for merging this :) !!! Thank you all for your awesome reviews. |
|
Can one of the admins verify this patch? |
What changes were proposed in this pull request?
We should not try casting everything returned by
removeLeadingZerosFromNumberTypePartitionto string, as it returns null value for the cases when partition has null value and is already replaced byDEFAULT_PARTITION_NAMEWhy are the changes needed?
for null partitions where
removeLeadingZerosFromNumberTypePartitionis called it would throw a NPE and hence the query would fail.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added a UT, which would fail with an NPE otherwise.