[SPARK-30262][SQL] Avoid NumberFormatException when totalSize is empty#26892
[SPARK-30262][SQL] Avoid NumberFormatException when totalSize is empty#26892southernriver wants to merge 1 commit intoapache:masterfrom
Conversation
|
Thank you for your first contribution, @southernriver . |
|
ok to test |
|
Test build #115338 has finished for PR 26892 at commit
|
| val totalSize = properties.get(StatsSetupConst.TOTAL_SIZE).filter(_.nonEmpty).map(BigInt(_)) | ||
| val rawDataSize = properties.get(StatsSetupConst.RAW_DATA_SIZE).filter(_.nonEmpty) | ||
| .map(BigInt(_)) | ||
| val rowCount = properties.get(StatsSetupConst.ROW_COUNT).filter(_.nonEmpty).map(BigInt(_)) |
There was a problem hiding this comment.
This is a logical patch, but there is no evidence when this happens.
Could you give me a reproducible procedure which you met?
There was a problem hiding this comment.
I'm wondering if you are hitting the other system's bug. Then, you had better fix the root cause.
This is a related to the robustness of the code and may lead to unexpected exception in some unpredictable situation.Here is the case:
There was a problem hiding this comment.
Also, can you add tests for that?
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Okay. Since this is a trivial safeguard, I'll merge this to master. Thank you, @southernriver , @srowen , @maropu .
|
@southernriver . You are added to the Apache Spark contributor group and SPARK-30262 is assigned to you. Thank you again. |
What changes were proposed in this pull request?
We could get the Partitions Statistics Info.But in some specail case, The Info like totalSize,rawDataSize,rowCount maybe empty. When we do some ddls like
desc formatted partition,the NumberFormatException is showed as below:Although we can use 'Analyze table partition ' to update the totalSize,rawDataSize or rowCount, it's unresonable for normal SQL to throw NumberFormatException for Empty totalSize.We should fix the empty case when readHiveStats.
Why are the changes needed?
This is a related to the robustness of the code and may lead to unexpected exception in some unpredictable situation.Here is the case:

Does this PR introduce any user-facing change?
No
How was this patch tested?
manual