Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug in temporal utilities due to DST being ignored. #955

Merged
merged 3 commits into from
Nov 20, 2021

Conversation

novemberkilo
Copy link
Contributor

Which issue does this PR close?

Closes #901

What changes are included in this PR?

Replaces the function using_chrono_tz with a function using_chrono_tz_and_utc_naive_datetime

The rationale for this is that an offset for a timezone depends on the point in time at which the offset is required -- it will vary depending on whether DST is in effect at that point in time or not.

Are there any user-facing changes?

I'm not sure. using_chrono_tz is a publicly available function (albeit under the chrono-tz feature flag) so strictly speaking the API is changing? If this is a concern, we could have both functions appear (with using_chrono_tz remaining unused).

What do you think @alamb ?

@github-actions github-actions bot added the arrow Changes to the arrow crate label Nov 18, 2021
@codecov-commenter
Copy link

codecov-commenter commented Nov 18, 2021

Codecov Report

Merging #955 (7e0e187) into master (32d8c0a) will increase coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #955      +/-   ##
==========================================
+ Coverage   82.31%   82.32%   +0.01%     
==========================================
  Files         168      168              
  Lines       48715    48713       -2     
==========================================
+ Hits        40100    40105       +5     
+ Misses       8615     8608       -7     
Impacted Files Coverage Δ
arrow/src/compute/kernels/temporal.rs 97.35% <100.00%> (+1.27%) ⬆️
parquet/src/encodings/encoding.rs 93.71% <0.00%> (+0.19%) ⬆️
arrow/src/array/transform/mod.rs 85.47% <0.00%> (+0.27%) ⬆️
parquet_derive/src/parquet_field.rs 66.43% <0.00%> (+0.45%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32d8c0a...7e0e187. Read the comment docs.

None => $builder.append_null()?,
match $array.value_as_datetime(i) {
Some(utc) => {
let fixed_offset = match parse(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As written, I think this code will try and parse $tz for each row in the array (even though the $tz is a constant).

I wonder if it would be possible to try and parse $tz once to figure out if it was fixed (e.g. a Some<FixedOffset>, and then only check if that was None for each row before computing using_chrono_tz_and_utc_naive_date_time

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up @alamb - I have addressed this in a subsequent commit.

@alamb
Copy link
Contributor

alamb commented Nov 18, 2021

Thank you @novemberkilo -- I appreciate the follow up

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good @novemberkilo -- thank you

@alamb alamb merged commit 9703f98 into apache:master Nov 20, 2021
alamb pushed a commit that referenced this pull request Nov 22, 2021
* Check behaviour of temporal utilities for DST.

* Fix temporal util bug ignoring dst.

* Refactor macro for efficiency.
alamb pushed a commit that referenced this pull request Nov 23, 2021
* Check behaviour of temporal utilities for DST.

* Fix temporal util bug ignoring dst.

* Refactor macro for efficiency.
alamb added a commit that referenced this pull request Nov 23, 2021
* Check behaviour of temporal utilities for DST.

* Fix temporal util bug ignoring dst.

* Refactor macro for efficiency.

Co-authored-by: Navin <navin@novemberkilo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Temporal utilities miss variations due to daylight savings
3 participants