Extend DateTimeFormatCheck to work for multiseries - #4300
Merged
Merged
Conversation
MichaelFu512
commented
Sep 7, 2023
| ... } | ||
| ... ] | ||
|
|
||
| The column "Weeks" has a date that does not follow the weekly pattern, which is considered misaligned. |
Contributor
Author
There was a problem hiding this comment.
Removed because it looks like an exact copy of line 156-200
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #4300 +/- ##
=======================================
+ Coverage 99.7% 99.7% +0.1%
=======================================
Files 357 357
Lines 39587 39694 +107
=======================================
+ Hits 39467 39574 +107
Misses 120 120
☔ View full report in Codecov by Sentry. |
MichaelFu512
commented
Sep 7, 2023
| if not pd.DatetimeIndex(no_nan_datetime_values).is_monotonic_increasing: | ||
| messages.append( | ||
| DataCheckError( | ||
| message="Datetime values must be sorted in ascending order.", |
Contributor
Author
There was a problem hiding this comment.
Unsure if I need to change the message here to say something like "Datetime values {at series x} must be sorted in ascending order" or if it's alright to just keep it general (note: this line is not currently in the for loop that loops through all the different series)
Contributor
There was a problem hiding this comment.
I think it's fine to keep it general here!
MichaelFu512
marked this pull request as ready for review
September 8, 2023 00:08
MichaelFu512
requested review from
christopherbunn,
eccabay and
jeremyliweishih
September 8, 2023 00:08
eccabay
suggested changes
Sep 11, 2023
eccabay
left a comment
eccabay
left a comment
Contributor
There was a problem hiding this comment.
Just a few nitpicks and some testing consolidation, but otherwise looks great! Thanks for doing this
| if not pd.DatetimeIndex(no_nan_datetime_values).is_monotonic_increasing: | ||
| messages.append( | ||
| DataCheckError( | ||
| message="Datetime values must be sorted in ascending order.", |
Contributor
There was a problem hiding this comment.
I think it's fine to keep it general here!
Comment on lines
-352
to
-380
| dates = ( | ||
| pd.date_range("2021-01-01", periods=15, freq="2D") | ||
| .drop("2021-01-13") | ||
| .append(pd.date_range("2021-01-30", periods=1)) | ||
| .append(pd.date_range("2021-01-31", periods=86, freq="2D")) | ||
| ) | ||
| X = pd.DataFrame({"dates": dates}, dtype="datetime64[ns]") | ||
|
|
||
| ww_payload = infer_frequency( | ||
| X["dates"], | ||
| debug=True, | ||
| window_length=WINDOW_LENGTH, | ||
| threshold=THRESHOLD, | ||
| ) | ||
|
|
||
| assert datetime_format_check.validate(X, y) == [ | ||
| DataCheckError( | ||
| message="Column 'dates' has datetime values missing between start and end date.", | ||
| data_check_name=datetime_format_check_name, | ||
| message_code=DataCheckMessageCode.DATETIME_IS_MISSING_VALUES, | ||
| ).to_dict(), | ||
| DataCheckError( | ||
| message="Column 'dates' has datetime values that do not align with the inferred frequency.", | ||
| data_check_name=datetime_format_check_name, | ||
| message_code=DataCheckMessageCode.DATETIME_HAS_MISALIGNED_VALUES, | ||
| ).to_dict(), | ||
| get_uneven_error("dates", ww_payload), | ||
| ] | ||
|
|
Contributor
Author
There was a problem hiding this comment.
It looks like an exact copy of line 323 - 349
eccabay
approved these changes
Sep 12, 2023
jeremyliweishih
approved these changes
Sep 12, 2023
jeremyliweishih
left a comment
jeremyliweishih
left a comment
Collaborator
There was a problem hiding this comment.
LGTM - just some punctuation requested.
Co-authored-by: Jeremy Shih <jeremyliweishih@gmail.com>
Co-authored-by: Jeremy Shih <jeremyliweishih@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request Description
Extend DateTimeFormatCheck to work for multiseries, works on stacked datasets.
Closes #4301
Small example of what this does
Say we had a multiseries problem with a stacked dataset, with two series (0, 1) and goes from 2021-01-01 to 2021-01-15.
For Series 1, the date "2021-01-07" is missing (line 14) while Series 0 has an extra date "2021-01-29" (line 31).
When we use the datacheck and run validate:
It gives the following result:
