fix(core): don't blank a datetime column when its format coerces every value to NaT#42405
fix(core): don't blank a datetime column when its format coerces every value to NaT#42405rusackas wants to merge 1 commit into
Conversation
…y value to NaT When a temporal column's declared python_date_format doesn't match its actual data, pd.to_datetime(..., errors='coerce') returns all-NaT and the result was assigned unconditionally, silently nulling the whole column. This surfaces in the Samples/drill-detail pane, where a chart's granularity column (e.g. an epoch-millis 'year' that inherited a '%Y' string format) renders as N/A for every row. Keep the original values when a format coerces every non-null entry to NaT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Code Review Agent Run #db7743Actionable Suggestions - 0Review Details
Bito Usage GuideCommands Type the following command in the pull request comment and save the comment.
Refer to the documentation for additional commands. Configuration This repository uses Documentation & Help AI Code Review powered by |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #42405 +/- ##
==========================================
- Coverage 65.24% 65.24% -0.01%
==========================================
Files 2795 2795
Lines 157643 157646 +3
Branches 36061 36062 +1
==========================================
+ Hits 102853 102854 +1
- Misses 52814 52815 +1
- Partials 1976 1977 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
2 participants
SUMMARY
When a temporal column's declared
python_date_formatdoesn't match its actual data,pd.to_datetime(..., errors="coerce")returns all-NaT, and_process_datetime_columnassigned that result unconditionally — silently nulling the entire column.This surfaces in the Samples / drill-detail pane: a chart whose granularity is a temporal column can send that column through normalization with a format that no longer fits the value it receives. Concretely,
video_game_sales.yearis an epoch-millis-backed column that inherits a%Ystring format;pd.to_datetime(1136073600000, format="%Y")→NaT, so everyyearvalue in the Samples tab renders as N/A, even though the chart itself renders (its SQL path passes the already-epoch value straight through). Any time-series chart on a dataset with a misfit temporal-column format hits this.The fix: when a format coerces every non-null value to
NaT, treat it as a mismatch and keep the original values rather than blanking the column. A format that successfully parses any value is applied exactly as before, so no currently-working case changes.BEFORE/AFTER
NaT→ Samples shows N/A for every row.TESTING INSTRUCTIONS
pytest tests/unit_tests/utils/test_core.py -k normalize_dttm— the newtest_normalize_dttm_col_mismatched_format_keeps_valuesfails before the fix (column blanked to NaT) and passes after; the existing normalize tests are unchanged.ADDITIONAL INFORMATION
🤖 Generated with Claude Code