-
-
Notifications
You must be signed in to change notification settings - Fork 17.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix wrong save of datetime64[s] in HDFStore #59018
Fix wrong save of datetime64[s] in HDFStore #59018
Conversation
Thanks for reviewing my code. I've accepted your suggestions and made new commits. Why did the CI checks fail? |
Might also close #57085? |
Will look into that. |
The code passed new tests but has regression failure on an old test case in Digging deeper, the current code base seems to be consistently using So the alternative I'm thinking is to do a forceful conversion on user input to nanosecond, or issue a warning when user tries to use other dtypes besides |
I see, thanks @jbrockmendel for linking this issue. If I understand the discussion correctly, non-nano resolutions are not yet fully integrated with the current code base, but changes are intended. And yes, this fix might also resolve 57085. I think I'll keep working to add support for non-nano time units in HDFStore. |
doc/source/whatsnew/v3.0.0.rst
Outdated
@@ -465,7 +465,7 @@ Timedelta | |||
|
|||
Timezones | |||
^^^^^^^^^ | |||
- | |||
- :meth:`io::pytables::_set_tz` was failing to parse datetime64 dtypes correctly (:issue:`59004`) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you use your older whatsnew note instead of this one? Also could you put this in the IO section?
@@ -345,3 +347,22 @@ def test_read_with_where_tz_aware_index(tmp_path, setup_path): | |||
store.append(key, expected, format="table", append=True) | |||
result = pd.read_hdf(path, key, where="DATE > 20151130") | |||
tm.assert_frame_equal(result, expected) | |||
|
|||
|
|||
def test_set_tz_datetime64_unit(tmp_path, setup_path): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you use your older test instead of this one? We prefer tests that exercise the public API
If you're up for it, I think the entire HDF code needs to be updated to support non-nanosecond resolutions i.e. not always coerce things to nanoseconds |
0e4abde
to
bf3ee62
Compare
I think the code is working now. The CI test xfails on I upgraded to
|
Thanks @chaoyihu |
doc/source/whatsnew/v2.2.2.rst
file if fixing a bug or adding a new feature.