[SPARK-21330][SQL] Bad partitioning does not allow to read a JDBC table with extreme values on the partition column #18800
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
srowen
reviewed
Aug 1, 2017
| @@ -64,7 +64,8 @@ private[sql] object JDBCRelation extends Logging { | |||
| s"bound. Lower bound: $lowerBound; Upper bound: $upperBound") | |||
|
|
|||
| val numPartitions = | |||
| if ((upperBound - lowerBound) >= partitioning.numPartitions) { | |||
| if ((upperBound - lowerBound) >= partitioning.numPartitions || | |||
| (upperBound - lowerBound) < 0) { | |||
There was a problem hiding this comment.
Looks good. for bonus points, add a comment about what this is for, and indent this line two more spaces
| val df = sql("SELECT * FROM partsoverflow") | ||
| checkNumPartitions(df, expectedNumPartitions = 3) | ||
| assert(df.collect().length == 3) | ||
|
|
There was a problem hiding this comment.
And maybe delete this blank for tidiness
|
Test build #80130 has finished for PR 18800 at commit
|
|
Test build #80131 has finished for PR 18800 at commit
|
|
Test build #3867 has finished for PR 18800 at commit
|
srowen
approved these changes
Aug 3, 2017
asfgit
pushed a commit
that referenced
this pull request
Aug 4, 2017
…le with extreme values on the partition column ## What changes were proposed in this pull request? An overflow of the difference of bounds on the partitioning column leads to no data being read. This patch checks for this overflow. ## How was this patch tested? New unit test. Author: Andrew Ray <ray.andrew@gmail.com> Closes #18800 from aray/SPARK-21330. (cherry picked from commit 25826c7) Signed-off-by: Sean Owen <sowen@cloudera.com>
asfgit
pushed a commit
that referenced
this pull request
Aug 4, 2017
…le with extreme values on the partition column ## What changes were proposed in this pull request? An overflow of the difference of bounds on the partitioning column leads to no data being read. This patch checks for this overflow. ## How was this patch tested? New unit test. Author: Andrew Ray <ray.andrew@gmail.com> Closes #18800 from aray/SPARK-21330. (cherry picked from commit 25826c7) Signed-off-by: Sean Owen <sowen@cloudera.com>
|
Merged to master/2.2/2.1 |
MatthewRBruce
pushed a commit
to Shopify/spark
that referenced
this pull request
Jul 31, 2018
…le with extreme values on the partition column ## What changes were proposed in this pull request? An overflow of the difference of bounds on the partitioning column leads to no data being read. This patch checks for this overflow. ## How was this patch tested? New unit test. Author: Andrew Ray <ray.andrew@gmail.com> Closes apache#18800 from aray/SPARK-21330. (cherry picked from commit 25826c7) Signed-off-by: Sean Owen <sowen@cloudera.com>
jzhuge
pushed a commit
to jzhuge/spark
that referenced
this pull request
Aug 20, 2018
…le with extreme values on the partition column An overflow of the difference of bounds on the partitioning column leads to no data being read. This patch checks for this overflow. New unit test. Author: Andrew Ray <ray.andrew@gmail.com> Closes apache#18800 from aray/SPARK-21330. (cherry picked from commit 25826c7) Signed-off-by: Sean Owen <sowen@cloudera.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
An overflow of the difference of bounds on the partitioning column leads to no data being read. This
patch checks for this overflow.
How was this patch tested?
New unit test.