[SPARK-54916][ML] Fix Parquet footer error in DecisionTree test suites by caching RDDs#54941
Open
azmatsiddique wants to merge 5 commits intoapache:masterfrom
Open
[SPARK-54916][ML] Fix Parquet footer error in DecisionTree test suites by caching RDDs#54941azmatsiddique wants to merge 5 commits intoapache:masterfrom
azmatsiddique wants to merge 5 commits intoapache:masterfrom
Conversation
…n last CSV column ### What changes were proposed in this pull request? This PR fixes an issue where the CSV reader inconsistently parses empty quoted strings (`""`) when the `escape` option is set to an empty string (`""`). Previously, mid-line empty quoted strings correctly resolved to null/empty, but the last column resolved to a literal `"` character due to univocity parser behavior. ### Why are the changes needed? To ensure consistent parsing of CSV data regardless of column position. ### Does this PR introduce _any_ user-facing change? Yes, it fixes a bug where users were receiving incorrect data (a literal quote instead of an empty/null value) for the last column in a row under specific CSV configurations. ### How was this patch tested? Added a new regression test in `CSVSuite` that verifies consistent parsing of both mid-line and end-of-line empty quoted fields.
azmatsiddique
force-pushed
the
fix/spark-54916-parquet-footer
branch
from
dcf3825 to
2ca1250
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR fixes a CANNOT_READ_FILE_FOOTER error encountered in DecisionTreeClassifierSuite and DecisionTreeRegressorSuite when running with Scala 2.13 in Spark 4.2.0-preview3.
The fix involves explicitly calling .cache() on the RDDs initialized in beforeAll() (and in TreeTests.getTreeReadWriteData). This forces the materialization of the underlying ArraySeq (introduced via .toImmutableArraySeq) before the test cases attempt to fit models, which involves writing temporary Parquet files.
Why are the changes needed?
Following the migration to Scala 2.13 collection conversions (specifically toImmutableArraySeq), certain ML test suites began failing with corrupted Parquet files.
The root cause appears to be a race condition or serialization issue where the lazy evaluation of the ImmutableArraySeq coincided with Spark's background Parquet writing during DecisionTree.fit. By caching the RDDs immediately after creation in beforeAll, we ensure the data is fully materialized and stable before any Parquet write operations begin.
Does this PR introduce any user-facing change?
No. This is a fix for internal test suites only.
How was this patch tested?
Verified locally by running the affected test suite: build/sbt "mllib/testOnly org.apache.spark.ml.classification.DecisionTreeClassifierSuite" The tests now pass consistently without Parquet footer errors.
Was this patch authored or co-authored using generative AI tooling?
No