BUG: fix TypeError for invalid integer dates %Y%m%d with errors='ignore' #26585

nathalier · 2019-05-31T02:06:51Z

closes to_datetime returns TypeError for invalid integer dates with %Y%m%d format #26583
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

array_strptime returned TypeError when trying to slice 'too long' integer for the given format %Y%m%d (for example 2121010101).
After parsing date in the first 8 symbols it tried to return the remaining symbols in ValueError message as a slice of integer which in turn caused TypeError.
Converted to string value is now used to make slice for that ValueError message.
In case of 20209911, it tried to parse 20209911 to datetime(2020, 9, 9) and had 11 left unparsed.

pep8speaks · 2019-05-31T02:06:56Z

Hello @nathalier! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-06-01 20:03:56 UTC

codecov · 2019-05-31T02:44:16Z

Codecov Report

Merging #26585 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26585      +/-   ##
==========================================
- Coverage   91.84%   91.83%   -0.01%     
==========================================
  Files         174      174              
  Lines       50644    50644              
==========================================
- Hits        46516    46511       -5     
- Misses       4128     4133       +5

Flag	Coverage Δ
#multiple	`90.37% <ø> (ø)`	⬆️
#single	`41.69% <ø> (-0.11%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`97% <0%> (-0.12%)`	⬇️
pandas/util/testing.py	`90.81% <0%> (-0.11%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c54dd2...c53b284. Read the comment docs.

codecov · 2019-05-31T02:44:16Z

Codecov Report

Merging #26585 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26585      +/-   ##
==========================================
- Coverage   91.84%   91.83%   -0.01%     
==========================================
  Files         174      174              
  Lines       50644    50644              
==========================================
- Hits        46515    46511       -4     
- Misses       4129     4133       +4

Flag	Coverage Δ
#multiple	`90.37% <ø> (ø)`	⬆️
#single	`41.69% <ø> (-0.09%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`97% <0%> (-0.12%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c6a7cc1...458b5cb. Read the comment docs.

mroeschke · 2019-05-31T15:10:38Z

pandas/_libs/tslibs/strptime.pyx

@@ -140,13 +140,13 @@ def array_strptime(object[:] values, object fmt,
                    iresult[i] = NPY_NAT
                    continue
                raise ValueError("time data %r does not match "
-                                 "format %r (match)" % (values[i], fmt))
+                                 "format %r (match)" % (val, fmt))


Do you mind converting these to use format?

I've just tried to do it but a couple of tests failed. I don't know the reason yet.

mroeschke · 2019-05-31T15:12:36Z

Minor comment, needs a whatsnew, but generally LGTM.

jreback · 2019-06-01T14:18:37Z

doc/source/whatsnew/v0.25.0.rst

@@ -427,6 +427,7 @@ Datetimelike
 - Bug in :class:`Series` and :class:`DataFrame` repr where ``np.datetime64('NaT')`` and ``np.timedelta64('NaT')`` with ``dtype=object`` would be represented as ``NaN`` (:issue:`25445`)
 - Bug in :func:`to_datetime` which does not replace the invalid argument with ``NaT`` when error is set to coerce (:issue:`26122`)
 - Bug in adding :class:`DateOffset` with nonzero month to :class:`DatetimeIndex` would raise ``ValueError`` (:issue:`26258`)
+- Bug in :func:`to_datetime` which raises a ``TypeError`` when called with a too long for a given format integer date with ``errors='ignore'``


can you say this is for %Y%m%d format

also not clear about 'too long', maybe length of the integer is != 8

jreback · 2019-06-01T14:18:58Z

pandas/tests/indexes/datetimes/test_tools.py

+        [20129930, 20129930],
+        [20120011, 20120011],
+        [2012010101, 2012010101]])
+    def test_int_to_datetime_format_YYYYMMDD_typeerror(self, int_date,


can you try with a too short as well?

Hi @jreback,
Thanks for feedback.
I reworded whatsnew message and added a couple of short dates.

Also I found result can be unpredictable for some short dates (either in integer or string format) with length > 4.

pd.to_datetime(12912, format="%Y%m%d", errors='ignore') Out[4]: 12912 pd.to_datetime(21213, format="%Y%m%d", errors='ignore') Out[5]: datetime.datetime(2, 12, 13, 0, 0)

Nevertheless it's not connected to this TypeError.
I'll add these examples to corresponding issue.

that is weird, these should for sure fail, but maybe going down an odd path. ok can address in another issue.

@jreback I added it to #14487. It's all about how format '%Y%m%d is treated.

…re' (# GH 26583) array_strptime returned TypeError when trying to slice 'too long' integer for the given format %Y%m%d (for example 2121010101). After parsing date in the first 8 symbols it tried to return the remaining symbols in ValueError message as a slice of integer which in turn caused TypeError. Converted to string value is now used to make slice for that ValueError message. In case of 20209911, it tried to parse 20209911 to datetime(2020, 9, 9) and had 11 unparsed.

jreback · 2019-06-02T23:52:21Z

@nathalier you are adding the short digits to an issue? (not this one that is being closed, correct)? if so we can merge this.

jorisvandenbossche · 2019-06-05T19:06:47Z

Thanks @nathalier !

…re' (# GH 26583) (pandas-dev#26585)

simonjayhawkins added Bug Datetime Datetime data dtype labels May 31, 2019

nathalier force-pushed the invalid-int-date-todatetime branch 2 times, most recently from 38a65a6 to 1e97e8b Compare May 31, 2019 11:15

nathalier changed the title ~~BUG: fix IndexError for invalid integer dates %Y%m%d with errors='ign…~~ BUG: fix TypeError for invalid integer dates %Y%m%d with errors='ign… May 31, 2019

mroeschke reviewed May 31, 2019

View reviewed changes

nathalier force-pushed the invalid-int-date-todatetime branch from 480df1b to 458b5cb Compare June 1, 2019 01:59

jreback requested changes Jun 1, 2019

View reviewed changes

nathalier force-pushed the invalid-int-date-todatetime branch from 458b5cb to d93d835 Compare June 1, 2019 19:51

nathalier force-pushed the invalid-int-date-todatetime branch from d93d835 to 11eddeb Compare June 1, 2019 20:03

jreback added this to the 0.25.0 milestone Jun 2, 2019

jreback approved these changes Jun 2, 2019

View reviewed changes

jorisvandenbossche changed the title ~~BUG: fix TypeError for invalid integer dates %Y%m%d with errors='ign…~~ BUG: fix TypeError for invalid integer dates %Y%m%d with errors='ignore' Jun 5, 2019

jorisvandenbossche approved these changes Jun 5, 2019

View reviewed changes

jorisvandenbossche merged commit 1d7ad5f into pandas-dev:master Jun 5, 2019

vaibhavhrt pushed a commit to vaibhavhrt/pandas that referenced this pull request Jun 6, 2019

BUG: fix TypeError for invalid integer dates %Y%m%d with errors='igno…

5271868

…re' (# GH 26583) (pandas-dev#26585)

topper-123 mentioned this pull request Jun 6, 2019

Test for tests\indexes\datetimes\test_tools.py fails #26696

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: fix TypeError for invalid integer dates %Y%m%d with errors='ignore' #26585

BUG: fix TypeError for invalid integer dates %Y%m%d with errors='ignore' #26585

nathalier commented May 31, 2019 •

edited

Loading

pep8speaks commented May 31, 2019 •

edited

Loading

codecov bot commented May 31, 2019

codecov bot commented May 31, 2019 •

edited

Loading

mroeschke May 31, 2019 •

edited

Loading

nathalier Jun 1, 2019

mroeschke commented May 31, 2019

jreback Jun 1, 2019

jreback Jun 1, 2019

nathalier Jun 1, 2019

jreback Jun 2, 2019

nathalier Jun 3, 2019 •

edited

Loading

jreback commented Jun 2, 2019

jorisvandenbossche commented Jun 5, 2019

BUG: fix TypeError for invalid integer dates %Y%m%d with errors='ignore' #26585

BUG: fix TypeError for invalid integer dates %Y%m%d with errors='ignore' #26585

Conversation

nathalier commented May 31, 2019 • edited Loading

pep8speaks commented May 31, 2019 • edited Loading

Comment last updated at 2019-06-01 20:03:56 UTC

codecov bot commented May 31, 2019

Codecov Report

codecov bot commented May 31, 2019 • edited Loading

Codecov Report

mroeschke May 31, 2019 • edited Loading

Choose a reason for hiding this comment

nathalier Jun 1, 2019

Choose a reason for hiding this comment

mroeschke commented May 31, 2019

jreback Jun 1, 2019

Choose a reason for hiding this comment

jreback Jun 1, 2019

Choose a reason for hiding this comment

nathalier Jun 1, 2019

Choose a reason for hiding this comment

jreback Jun 2, 2019

Choose a reason for hiding this comment

nathalier Jun 3, 2019 • edited Loading

Choose a reason for hiding this comment

jreback commented Jun 2, 2019

jorisvandenbossche commented Jun 5, 2019

nathalier commented May 31, 2019 •

edited

Loading

pep8speaks commented May 31, 2019 •

edited

Loading

codecov bot commented May 31, 2019 •

edited

Loading

mroeschke May 31, 2019 •

edited

Loading

nathalier Jun 3, 2019 •

edited

Loading