Improved datetime format inference - #543
Merged
Merged
Conversation
LilianBoulard
left a comment
Member
There was a problem hiding this comment.
Thanks for the PR, a few tweaks to merge and it's good for me!
Member
There was a problem hiding this comment.
Maybe a remark on the _infer_date_format function: since it's only used in the TableVectorizer, I think it should be moved over to its file.
LilianBoulard
left a comment
Member
There was a problem hiding this comment.
Messed up something in my previous review, this is a fix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #540
The bug in #540 was related to pandas 2.0 update: as datetime format inference became stricter, a bug which was silent (inconsistent date format inferred for one column in the test) became loud.
This PR fixes this and improves date format detection (and thus datetime column detection in
TableVectorizer). This replaces usingpd.datetimedirectly for datetime format inference.To do this, we try pandas's
guess_datetime_format(on a subset) both withdayfirstbeingTrueandFalse, and see if one of these options finds a single format for all rows. If both work, we return the monthfirst format and raise a warning. If both return multiple formats, we give up.This finds the right %d-%m-%Y format for the failing test example, instead of failing or returning mixed format (see below).
pd.to_datetimeformat inferenceIf I understand correctly, pandas datetime format inference (in
pd.to_datetime) works like this:Versions < 2.0
If not
infer_datetime_format, the format of each row is inferred independently.If
infer_datetime_format, the format is inferred from the first non-missing example, and pandas tries to apply it to the other example (but can use another format otherwise).This can easily create issues, for instance in our tests:
For all rows, the inferred format was %m-%d-%Y expect 13-02-2000, for which the format was %d-%m-%Y.
Version >= 2.0
The format is inferred from the first non-missing example, and pandas tries to apply it to the other example (and raise an error otherwise).