BUG: Catch overflow in both directions for checked add #14453

Merged
merged 1 commit into from Oct 22, 2016

Conversation

Projects
None yet
4 participants
Member

gfyoung commented Oct 19, 2016

Port of #14324 after project name change.

Member

gfyoung commented Oct 19, 2016 edited

@jreback : Doesn't seem like Appveyor is picking up this PR.

codecov-io commented Oct 19, 2016 edited

Current coverage is 85.25% (diff: 68.75%)

Merging #14453 into master will decrease coverage by <.01%

@@             master     #14453   diff @@
==========================================
  Files           140        140          
  Lines         50640      50654    +14   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43175      43184     +9   
- Misses         7465       7470     +5   
  Partials          0          0          

Powered by Codecov. Last update 65362aa...2dc49e4

jreback added this to the 0.19.1 milestone Oct 20, 2016

Contributor

jreback commented Oct 20, 2016

lgtm. can you add a whatsnew note. put in a new API Changes section in 0.19.1

Owner

jorisvandenbossche commented Oct 20, 2016 edited

Please no api change section for 0.19.1. We put it in the bug section for 0.19.0, so then can put it here there as well I think (we maybe should have put it in the api changes section for 0.19.0 ..)

@gfyoung gfyoung BUG: Catch overflow in both directions for checked add
1) Add checks to ensure that add overflow does not
   occur both in the positive or negative directions.
2) Add benchmarks to ensure that operations involving
   this checked add function are significantly impacted.
2dc49e4
Contributor

jreback commented Oct 20, 2016

bug fix section is fine

@jorisvandenbossche jorisvandenbossche merged commit 83a380c into pandas-dev:master Oct 22, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@gfyoung Thanks!

gfyoung deleted the gfyoung:add-overflow-bug branch Oct 22, 2016

Member

gfyoung commented Oct 24, 2016

@jreback , @jorisvandenbossche : So while it's good that we're catching underflow, when I try to integrate this check with tdi + tdi OR tdi + td, we run into issues:

>>> import pandas as pd
>>>
>>> tdi = pd.TimedeltaIndex([pd.NaT])
>>> td = pd.Timedelta('1 days')
>>>
>>> tdi - td   # Should return TimedeltaIndex([pd.NaT])
...
OverflowError: Overflow in int64 addition

This is because we are doing the checked add with the asi8 values, meaning that NaT becomes np.iinfo(np.int64).min, so the subtraction underflows as expected but not as we want. Not sure what the best way is to NOT do checked add for those elements whose value is NaN. Thoughts?

Additionally mask for that in the checking code?

Member

gfyoung commented Oct 24, 2016

Hmmm...I suppose, though that will impact performance again 😢 . I can give it a shot anyways and see what happens.

@jorisvandenbossche jorisvandenbossche added a commit to jorisvandenbossche/pandas that referenced this pull request Nov 2, 2016

@gfyoung @jorisvandenbossche gfyoung + jorisvandenbossche [Backport 14453] BUG: Catch overflow in both directions for checked a…
…dd (#14453)

1) Add checks to ensure that add overflow does not
   occur both in the positive or negative directions.
2) Add benchmarks to ensure that operations involving
   this checked add function are significantly impacted.
(cherry picked from commit 83a380c)
c93ba9f

@amolkahat amolkahat added a commit to amolkahat/pandas that referenced this pull request Nov 26, 2016

@gfyoung @amolkahat gfyoung + amolkahat BUG: Catch overflow in both directions for checked add (#14453)
1) Add checks to ensure that add overflow does not
   occur both in the positive or negative directions.
2) Add benchmarks to ensure that operations involving
   this checked add function are significantly impacted.
3b4c25b

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 7, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and TimeDelta). Follow-up to gh-14453.
8e21e60

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 7, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.
46909f0

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 10, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.
ce48455

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 11, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.
7990eec

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 15, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.
e57f3bb

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 15, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.
d238fbb

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 15, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.
59ba12b

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 15, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.
4f8a12d

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 15, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.

In addition, move checked add function to core/algorithms.
7a3e54e

@gfyoung gfyoung added a commit to gfyoung/pandas that referenced this pull request Dec 17, 2016

@gfyoung gfyoung BUG: Prevent addition overflow with TimedeltaIndex
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.

In addition, move checked add function to core/algorithms.
a086db6

@jorisvandenbossche jorisvandenbossche added a commit that referenced this pull request Dec 17, 2016

@gfyoung @jorisvandenbossche gfyoung + jorisvandenbossche BUG: Prevent addition overflow with TimedeltaIndex (#14816)
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.

In addition, move checked add function to core/algorithms.
bdbebc4

@ischurov ischurov added a commit to ischurov/pandas that referenced this pull request Dec 19, 2016

@gfyoung @ischurov gfyoung + ischurov BUG: Prevent addition overflow with TimedeltaIndex (#14816)
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.

In addition, move checked add function to core/algorithms.
ac5ec9d

@ShaharBental ShaharBental added a commit to ShaharBental/pandas that referenced this pull request Dec 26, 2016

@gfyoung @ShaharBental gfyoung + ShaharBental BUG: Prevent addition overflow with TimedeltaIndex (#14816)
Expands checked-add array addition introduced in
gh-14237 to include all other addition cases (i.e.
TimedeltaIndex and Timedelta). Follow-up to gh-14453.

In addition, move checked add function to core/algorithms.
3398529
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment