BUG: avoid overflow in Bday generate_range, closes #24252 #26651

jbrockmendel · 2019-06-04T21:05:19Z

closes BUG: pd.date_range gives wrong overflow error type when freq='B' #24252
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

codecov · 2019-06-04T21:40:08Z

Codecov Report

Merging #26651 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #26651      +/-   ##
==========================================
- Coverage   91.87%   91.87%   -0.01%     
==========================================
  Files         174      174              
  Lines       50692    50696       +4     
==========================================
+ Hits        46575    46576       +1     
- Misses       4117     4120       +3

Flag	Coverage Δ
#multiple	`90.41% <100%> (ø)`	⬆️
#single	`41.8% <50%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/tseries/offsets.py	`96.7% <100%> (+0.01%)`	⬆️
pandas/io/gbq.py	`78.94% <0%> (-10.53%)`	⬇️
pandas/core/frame.py	`97% <0%> (-0.12%)`	⬇️
pandas/util/testing.py	`90.91% <0%> (+0.1%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c07d71d...e200f32. Read the comment docs.

codecov · 2019-06-04T21:40:19Z

Codecov Report

Merging #26651 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #26651      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.02%     
==========================================
  Files         180      180              
  Lines       50746    50751       +5     
==========================================
- Hits        46623    46621       -2     
- Misses       4123     4130       +7

Flag	Coverage Δ
#multiple	`90.45% <100%> (-0.01%)`	⬇️
#single	`41.09% <33.33%> (-0.1%)`	⬇️

Impacted Files	Coverage Δ
pandas/tseries/offsets.py	`96.36% <100%> (-0.33%)`	⬇️
pandas/io/gbq.py	`88.88% <0%> (-11.12%)`	⬇️
pandas/core/frame.py	`96.89% <0%> (-0.12%)`	⬇️
pandas/util/testing.py	`90.94% <0%> (+0.1%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cfd65e9...b06440c. Read the comment docs.

topper-123 · 2019-06-04T21:59:35Z

This doesn't solve the overflow type:

>>> date = pd.Timestamp.max.floor("D").to_pydatetime().date()  # datetime.date(2262, 4, 11)
>>> freq = 'B'
>>> pd.date_range(date, periods=2, freq=freq)  # see periods=2
OverflowError: int too big to convert

I would guess this should be a OutOfBoundsDatetime and not a OverflowError?

jbrockmendel · 2019-06-04T23:37:13Z

Yah, it should probably be an OutOfBoundsDatetime instead of OverflowError. Will update.

pandas/tseries/offsets.py

jreback · 2019-06-06T00:29:47Z

pandas/_libs/tslibs/timestamps.pyx

-        ts = convert_to_tsobject(ts_input, tz, unit, 0, 0, nanosecond or 0)
+        try:
+            ts = convert_to_tsobject(ts_input, tz, unit, 0, 0, nanosecond or 0)
+        except OverflowError:


can you update the doc-string of convert_ot_tsobject. Why are we not catching the OverflowError inside?

I put this here because I wanted to do this error-catching in python-space instead of c-space so that convert_to_tsobject could have better optimizations, but it looks like convert_to_tsobject already has some raising cases, so this can be moved there. will update.

jreback · 2019-06-08T21:06:45Z

lgtm. ping when green.

jreback · 2019-06-10T21:56:33Z

code & tests lgtm. can you add a whatsnew. ping on green.

jbrockmendel · 2019-06-18T22:26:07Z

ping

jreback

small comment, otherwise lgtm.

jreback · 2019-06-18T23:58:21Z

pandas/_libs/tslibs/timestamps.pyx

@@ -24,6 +24,7 @@ from pandas._libs.tslibs.conversion cimport (
 from pandas._libs.tslibs.nattype cimport NPY_NAT, c_NaT as NaT
 from pandas._libs.tslibs.np_datetime cimport (
    check_dts_bounds, npy_datetimestruct, dt64_to_dtstruct)
+from pandas._libs.tslibs.np_datetime import OutOfBoundsDatetime


is this used here?

Nope, removed.

I'm holding out hope that one day we'll be able to lint these files.

jbrockmendel · 2019-06-19T20:29:00Z

In one Travis build I'm seeing an error I also get locally on many branches:

_______ TestDataFrameConstructors.test_constructor_maskedrecarray_dtype ________
[gw1] linux -- Python 3.7.3 /home/travis/miniconda3/envs/pandas-dev/bin/python
self = <pandas.tests.frame.test_constructors.TestDataFrameConstructors object at 0x7fa6ace49f98>
    def test_constructor_maskedrecarray_dtype(self):
        # Ensure constructor honors dtype
        data = np.ma.array(
            np.ma.zeros(5, dtype=[('date', '<f8'), ('price', '<f8')]),
            mask=[False] * 5)
>       data = data.view(ma.mrecords.mrecarray)
E       AttributeError: module 'numpy.ma' has no attribute 'mrecords'

Any idea what this is all about?

TomAugspurger · 2019-06-19T20:41:42Z

Not sure, but numpy.ma.mrecords doesn't seem to be imported by default. Would need import numpy.ma.mrecords before using ma.mrecords.

jbrockmendel · 2019-06-20T01:23:39Z

Looks like we import ma.mrecords as a runtime import in the DataFrame constructor, so this ma.mrecords will not exist if we run these tests before that runtime import gets hit. (I'm guessing we started randomizing test order recently?). Just added the import in the appropriate place in the test so it shouldn't matter anymore.

TomAugspurger · 2019-06-20T04:16:41Z

pandas/_libs/tslibs/conversion.pyx

+            ts = <int64_t>ts
+        except OverflowError:
+            # GH#26651 re-raise as OutOfBoundsDatetime
+            raise OutOfBoundsDatetime


Is including the bad ts in the exception message useful here? I imagine it's good for Timestamp(too_big_int), but not so much for other inputs (if they hit this).

jreback · 2019-06-21T02:02:07Z

thanks @jbrockmendel nice patch!

BUG: avoid overflow in Bday generate_range, closes pandas-dev#24252

e200f32

jbrockmendel mentioned this pull request Jun 4, 2019

BUG: pd.date_range gives wrong overflow error type when freq='B' #24252

Closed

gfyoung added Error Reporting Incorrect or improved errors from pandas Frequency DateOffsets Timezones Timezone data dtype labels Jun 4, 2019

update tests to check for OutOfBoundsDatetime instead of OverflowError

d387afe

jreback added this to the 0.25.0 milestone Jun 6, 2019

jreback requested changes Jun 6, 2019

View reviewed changes

jbrockmendel added 2 commits June 7, 2019 10:51

Merge branch 'master' of https://github.com/pandas-dev/pandas into oob

b4a296b

Move checking to convert_to_tsobject

68ba8a5

jbrockmendel added 2 commits June 10, 2019 10:15

Merge branch 'master' of https://github.com/pandas-dev/pandas into oob

0c4bf2c

remove message from preliminary scalar analogue test

595f0ec

jbrockmendel added 2 commits June 18, 2019 13:31

Merge branch 'master' of https://github.com/pandas-dev/pandas into oob

6cdfa91

whatsnew

bd79274

jreback reviewed Jun 18, 2019

View reviewed changes

jbrockmendel added 2 commits June 18, 2019 18:17

remove unused import

7a3f83e

Merge branch 'master' of https://github.com/pandas-dev/pandas into oob

ddd7f27

jbrockmendel added 2 commits June 19, 2019 18:19

Merge branch 'master' of https://github.com/pandas-dev/pandas into oob

105cd6b

add missing import

fe7c961

TomAugspurger reviewed Jun 20, 2019

View reviewed changes

TomAugspurger approved these changes Jun 20, 2019

View reviewed changes

jbrockmendel added 2 commits June 20, 2019 07:12

use ts in exception message

3d1503f

Merge branch 'master' of https://github.com/pandas-dev/pandas into oob

b06440c

jreback approved these changes Jun 21, 2019

View reviewed changes

jreback merged commit 388d22c into pandas-dev:master Jun 21, 2019

jbrockmendel deleted the oob branch June 21, 2019 02:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: avoid overflow in Bday generate_range, closes #24252 #26651

BUG: avoid overflow in Bday generate_range, closes #24252 #26651

jbrockmendel commented Jun 4, 2019

codecov bot commented Jun 4, 2019

codecov bot commented Jun 4, 2019 •

edited

Loading

topper-123 commented Jun 4, 2019

jbrockmendel commented Jun 4, 2019

jreback Jun 6, 2019

jbrockmendel Jun 7, 2019

jreback commented Jun 8, 2019

jreback commented Jun 10, 2019

jbrockmendel commented Jun 18, 2019

jreback left a comment

jreback Jun 18, 2019

jbrockmendel Jun 19, 2019

jbrockmendel commented Jun 19, 2019

TomAugspurger commented Jun 19, 2019

jbrockmendel commented Jun 20, 2019

TomAugspurger Jun 20, 2019 •

edited

Loading

jreback commented Jun 21, 2019

BUG: avoid overflow in Bday generate_range, closes #24252 #26651

BUG: avoid overflow in Bday generate_range, closes #24252 #26651

Conversation

jbrockmendel commented Jun 4, 2019

codecov bot commented Jun 4, 2019

Codecov Report

codecov bot commented Jun 4, 2019 • edited Loading

Codecov Report

topper-123 commented Jun 4, 2019

jbrockmendel commented Jun 4, 2019

jreback Jun 6, 2019

Choose a reason for hiding this comment

jbrockmendel Jun 7, 2019

Choose a reason for hiding this comment

jreback commented Jun 8, 2019

jreback commented Jun 10, 2019

jbrockmendel commented Jun 18, 2019

jreback left a comment

Choose a reason for hiding this comment

jreback Jun 18, 2019

Choose a reason for hiding this comment

jbrockmendel Jun 19, 2019

Choose a reason for hiding this comment

jbrockmendel commented Jun 19, 2019

TomAugspurger commented Jun 19, 2019

jbrockmendel commented Jun 20, 2019

TomAugspurger Jun 20, 2019 • edited Loading

Choose a reason for hiding this comment

jreback commented Jun 21, 2019

codecov bot commented Jun 4, 2019 •

edited

Loading

TomAugspurger Jun 20, 2019 •

edited

Loading