[SPARK-27100][SQL][2.4] Use Array instead of Seq in FilePartition to prevent StackOverflowError
#24957
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… prevent `StackOverflowError ` ShuffleMapTask's partition field is a FilePartition and FilePartition's 'files' field is a Stream$cons which is essentially a linked list. It is therefore serialized recursively. If the number of files in each partition is, say, 10000 files, recursing into a linked list of length 10000 overflows the stack The problem is only in Bucketed partitions. The corresponding implementation for non Bucketed partitions uses a StreamBuffer. The proposed change applies the same for Bucketed partitions. Existing unit tests. Added new unit test. The unit test fails without the patch. Manual testing on dataset used to reproduce the problem. Closes apache#24865 from parthchandra/SPARK-27100. Lead-authored-by: Parth Chandra <parthc@apple.com> Co-authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: DB Tsai <d_tsai@apple.com>
|
@parthchandra can you open a PR against the current master? |
|
This is a backport of 5a7aa6f . :) |
dongjoon-hyun
changed the title
Array instead of Seq in FilePartition to…Array instead of Seq in FilePartition to…
| * A collection of file blocks that should be read as a single task | ||
| * (possibly from multiple partitioned directories). | ||
| */ | ||
| case class FilePartition(index: Int, files: Array[PartitionedFile]) |
There was a problem hiding this comment.
It seems that this PR unexpectedly has a part of [SPARK-23817][SQL] Create file source V2 framework and migrate ORC read path. Could you try to remove this change about FilePartition?
cc @dbtsai , @gatorsmile
|
Test build #106859 has finished for PR 24957 at commit
|
dbtsai
reviewed
Jun 25, 2019
| * A collection of file blocks that should be read as a single task | ||
| * (possibly from multiple partitioned directories). | ||
| */ | ||
| case class FilePartition(index: Int, files: Seq[PartitionedFile]) extends RDDPartition |
There was a problem hiding this comment.
Do we need to have those changes as @dongjoon-hyun suggested? Why not just change Seq to Array here?
|
@dongjoon-hyun, @dbtsai, you guys are right. I included the FilePartition change as part of resolving the merge conflicts. But I can just make the change that DBTsai has suggested. I'll make the change, re-run the tests and update this PR. |
|
Updated |
gatorsmile
changed the title
Array instead of Seq in FilePartition to…Array instead of Seq in FilePartition to prevent StackOverflowError
gatorsmile
reviewed
Jun 25, 2019
| // large. tests for the condition where the serialization of such a task may result in a stack | ||
| // overflow if the files list is stored in a recursive data structure | ||
| // This test is ignored because it takes long to run (~3 min) | ||
| ignore("SPARK-27100 stack overflow: read data with large partitions") { |
There was a problem hiding this comment.
trigger the test and see whether it passes in 2.4?
There was a problem hiding this comment.
@parthchandra can you confirm this? Thanks.
There was a problem hiding this comment.
Yes. The test passes in 2.4 for both bucketing with/without hive.
|
Test build #106903 has finished for PR 24957 at commit
|
dbtsai
pushed a commit
that referenced
this pull request
Jun 26, 2019
…n` to prevent StackOverflowError … prevent `StackOverflowError ` ShuffleMapTask's partition field is a FilePartition and FilePartition's 'files' field is a Stream$cons which is essentially a linked list. It is therefore serialized recursively. If the number of files in each partition is, say, 10000 files, recursing into a linked list of length 10000 overflows the stack The problem is only in Bucketed partitions. The corresponding implementation for non Bucketed partitions uses a StreamBuffer. The proposed change applies the same for Bucketed partitions. Existing unit tests. Added new unit test. The unit test fails without the patch. Manual testing on dataset used to reproduce the problem. Closes #24957 from parthchandra/branch-2.4. Authored-by: Parth Chandra <parthc@apple.com> Signed-off-by: DB Tsai <d_tsai@apple.com>
|
Thanks. Merged into branch 2.4. |
rluta
pushed a commit
to rluta/spark
that referenced
this pull request
Sep 17, 2019
…n` to prevent StackOverflowError … prevent `StackOverflowError ` ShuffleMapTask's partition field is a FilePartition and FilePartition's 'files' field is a Stream$cons which is essentially a linked list. It is therefore serialized recursively. If the number of files in each partition is, say, 10000 files, recursing into a linked list of length 10000 overflows the stack The problem is only in Bucketed partitions. The corresponding implementation for non Bucketed partitions uses a StreamBuffer. The proposed change applies the same for Bucketed partitions. Existing unit tests. Added new unit test. The unit test fails without the patch. Manual testing on dataset used to reproduce the problem. Closes apache#24957 from parthchandra/branch-2.4. Authored-by: Parth Chandra <parthc@apple.com> Signed-off-by: DB Tsai <d_tsai@apple.com>
kai-chi
pushed a commit
to kai-chi/spark
that referenced
this pull request
Sep 26, 2019
…n` to prevent StackOverflowError … prevent `StackOverflowError ` ShuffleMapTask's partition field is a FilePartition and FilePartition's 'files' field is a Stream$cons which is essentially a linked list. It is therefore serialized recursively. If the number of files in each partition is, say, 10000 files, recursing into a linked list of length 10000 overflows the stack The problem is only in Bucketed partitions. The corresponding implementation for non Bucketed partitions uses a StreamBuffer. The proposed change applies the same for Bucketed partitions. Existing unit tests. Added new unit test. The unit test fails without the patch. Manual testing on dataset used to reproduce the problem. Closes apache#24957 from parthchandra/branch-2.4. Authored-by: Parth Chandra <parthc@apple.com> Signed-off-by: DB Tsai <d_tsai@apple.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… prevent
StackOverflowErrorShuffleMapTask's partition field is a FilePartition and FilePartition's 'files' field is a Stream$cons which is essentially a linked list. It is therefore serialized recursively.
If the number of files in each partition is, say, 10000 files, recursing into a linked list of length 10000 overflows the stack
The problem is only in Bucketed partitions. The corresponding implementation for non Bucketed partitions uses a StreamBuffer. The proposed change applies the same for Bucketed partitions.
Existing unit tests. Added new unit test. The unit test fails without the patch. Manual testing on dataset used to reproduce the problem.