-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-3930] [SPARK-3933] Support fixed-precision decimal in SQL, and some optimizations #2983
Conversation
Test build #22389 has started for PR 2983 at commit
|
Test build #22389 has finished for PR 2983 at commit
|
Test FAILed. |
Test build #22396 has started for PR 2983 at commit
|
Test build #22396 has finished for PR 2983 at commit
|
Test FAILed. |
Test build #22406 has started for PR 2983 at commit
|
I've marked this as not WIP anymore, because the main TODOs left are in the Hive support. I intend to send that as a separate patch, though I can also add it here. Right now this makes each Hive type be the unlimited-precision decimal, whereas in fact we should respect the precision and scale set in the Hive metastore in Hive 13; but the previous Spark SQL code doesn't respect that either so these are not a regression. |
Test build #22409 has started for PR 2983 at commit
|
Test build #22406 has finished for PR 2983 at commit
|
Test FAILed. |
Test build #22409 has finished for PR 2983 at commit
|
Test FAILed. |
Jenkins, test this please |
Test build #22414 has started for PR 2983 at commit
|
Test build #22414 has finished for PR 2983 at commit
|
Test FAILed. |
Test build #22418 has started for PR 2983 at commit
|
Test build #22418 has finished for PR 2983 at commit
|
Test FAILed. |
Jenkins, test this please |
Test build #22443 has started for PR 2983 at commit
|
Test build #22443 has finished for PR 2983 at commit
|
Test FAILed. |
Jenkins, test this please |
Test build #22459 has started for PR 2983 at commit
|
Test build #22459 has finished for PR 2983 at commit
|
Test FAILed. |
Test PASSed. |
| "TimestampType" ^^^ TimestampType | ||
) | ||
|
||
protected lazy val fixedDecimalType: Parser[DataType] = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is technically not required. This parser is only for reading data from old parquet files that were encoded with old versions of spark sql. Hopefully we can drop it completely some day.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see, wouldn't it be less confusing to leave it in for now though? I can also remove it if you prefer.
Hey Matei, this is pretty awesome! A few minor comments and this needs to be merged. Otherwise LGTM. |
…r now Implement CAST to fixed-precision decimal Add rules for propagating precision through decimal calculations These work by casting things to Decimal.Unlimited to do the actual operation, then adding a cast on the result. They will result in more casts than needed, but on the other hand they avoid having each arithmetic operator know about decimal precision rules. We might be able to add more rules later to eliminate some intermediate casts.
Optimize sums and averages on fixed-precision Decimals
Test build #22705 has started for PR 2983 at commit
|
Test build #22705 has finished for PR 2983 at commit
|
Test FAILed. |
Test build #504 has started for PR 2983 at commit
|
Test build #504 has finished for PR 2983 at commit
|
This is still marked WIP because there are a few TODOs, but I'll remove that tag when done.