ARROW-10947: [Rust][DataFusion] Optimize UTF8 to Date32 Conversion #8943
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Dandandan
approved these changes
Dec 16, 2020
Codecov Report
@@ Coverage Diff @@
## master #8943 +/- ##
=======================================
Coverage 83.26% 83.27%
=======================================
Files 195 195
Lines 48066 48062 -4
=======================================
- Hits 40024 40022 -2
+ Misses 8042 8040 -2
Continue to review full report at Codecov.
|
jorgecarleitao
approved these changes
Dec 21, 2020
GeorgeAp
pushed a commit
to sirensolutions/arrow
that referenced
this pull request
Jun 7, 2021
After adding benchmarking capability to the UTF8 to Date32/Date64 CAST functions there was opportunity to improve the performance. This PR uses inbuilt `chrono` functionality to calculate the number of days since CE then uses a constant to calculate the offset days relative to 1970-01-01. This improves performance around 10% for this operation relative to the `since` function presumably as `chrono` does not have to ensure the `from_ymd` is a valid date. Before: ``` cast utf8 to date32 512 time: [41.966 us 42.508 us 43.087 us] cast utf8 to date32 512 time: [40.591 us 40.661 us 40.740 us] cast utf8 to date32 512 time: [40.825 us 40.878 us 40.916 us] ``` After: ``` cast utf8 to date32 512 time: [36.557 us 36.839 us 37.200 us] cast utf8 to date32 512 time: [35.997 us 36.442 us 36.919 us] cast utf8 to date32 512 time: [35.750 us 35.969 us 36.160 us] ``` Closes apache#8943 from seddonm1/utf8-date32-optimize Authored-by: Mike Seddon <seddonm1@gmail.com> Signed-off-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After adding benchmarking capability to the UTF8 to Date32/Date64 CAST functions there was opportunity to improve the performance.
This PR uses inbuilt
chronofunctionality to calculate the number of days since CE then uses a constant to calculate the offset days relative to 1970-01-01. This improves performance around 10% for this operation relative to thesincefunction presumably aschronodoes not have to ensure thefrom_ymdis a valid date.Before:
After: