New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Masking and overflow checks for datetimeindex and timedeltaindex ops #18020

Merged
merged 17 commits into from Nov 4, 2017

Conversation

Projects
None yet
5 participants
@jbrockmendel
Member

jbrockmendel commented Oct 29, 2017

There are a bunch of new tests (not obvious what the appropriate place is for a WIP test matrix like this, pls advise). The ones that will fail under master are test_timedeltaindex_add_timestamp_nat_masking and test_datetimeindex_sub_timestamp_overflow

Start filling out a test matrix of arithmetic ops.

  • closes #17991
  • tests added / passed
  • passes git diff upstream/master -u -- "*.py" | flake8 --diff
  • whatsnew entry

jbrockmendel added some commits Oct 29, 2017

masking and overflow checks for datetimeindex and timedeltaindex ops
WIP filling out a test matrix of arithmetic ops

closes #17991
Show outdated Hide outdated pandas/tests/indexes/datetimelike.py Outdated
@codecov

This comment has been minimized.

Show comment
Hide comment
@codecov

codecov bot Oct 29, 2017

Codecov Report

Merging #18020 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #18020      +/-   ##
==========================================
- Coverage   91.25%   91.23%   -0.02%     
==========================================
  Files         163      163              
  Lines       50123    50124       +1     
==========================================
- Hits        45741    45733       -8     
- Misses       4382     4391       +9
Flag Coverage Δ
#multiple 89.05% <100%> (ø) ⬆️
#single 40.32% <20%> (-0.06%) ⬇️
Impacted Files Coverage Δ
pandas/core/indexes/timedeltas.py 91.19% <100%> (ø) ⬆️
pandas/core/indexes/datetimelike.py 97.1% <100%> (ø) ⬆️
pandas/core/indexes/datetimes.py 95.51% <100%> (ø) ⬆️
pandas/io/gbq.py 25% <0%> (-58.34%) ⬇️
pandas/core/frame.py 97.75% <0%> (-0.1%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3268f4e...4dbdfd7. Read the comment docs.

codecov bot commented Oct 29, 2017

Codecov Report

Merging #18020 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #18020      +/-   ##
==========================================
- Coverage   91.25%   91.23%   -0.02%     
==========================================
  Files         163      163              
  Lines       50123    50124       +1     
==========================================
- Hits        45741    45733       -8     
- Misses       4382     4391       +9
Flag Coverage Δ
#multiple 89.05% <100%> (ø) ⬆️
#single 40.32% <20%> (-0.06%) ⬇️
Impacted Files Coverage Δ
pandas/core/indexes/timedeltas.py 91.19% <100%> (ø) ⬆️
pandas/core/indexes/datetimelike.py 97.1% <100%> (ø) ⬆️
pandas/core/indexes/datetimes.py 95.51% <100%> (ø) ⬆️
pandas/io/gbq.py 25% <0%> (-58.34%) ⬇️
pandas/core/frame.py 97.75% <0%> (-0.1%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3268f4e...4dbdfd7. Read the comment docs.

@jreback

see comments

@jbrockmendel

This comment has been minimized.

Show comment
Hide comment
@jbrockmendel

jbrockmendel Oct 29, 2017

Member

OK, just removed the new file, took the two tests that currently fail on master and moved them into existing datetime and timedelta test files.

Member

jbrockmendel commented Oct 29, 2017

OK, just removed the new file, took the two tests that currently fail on master and moved them into existing datetime and timedelta test files.

jbrockmendel added some commits Oct 29, 2017

@jbrockmendel jbrockmendel referenced this pull request Oct 30, 2017

Closed

duplicate tests #18026

@jorisvandenbossche

This comment has been minimized.

Show comment
Hide comment
@jorisvandenbossche

jorisvandenbossche Oct 30, 2017

Member

Does this also fix #7996 ?

Member

jorisvandenbossche commented Oct 30, 2017

Does this also fix #7996 ?

@jbrockmendel

This comment has been minimized.

Show comment
Hide comment
@jbrockmendel

jbrockmendel Oct 30, 2017

Member

Does this also fix #7996 ?

No, but it does fix a related bug that probably belongs in the same issue

In [3]: s = pd.Series(pd.date_range('20130101',periods=3))
In [4]: dti = pd.DatetimeIndex(s)

In [5]: s-np.datetime64('2013-01-01')
Out[5]: 
0   15705 days 23:59:59.999984
1   15706 days 23:59:59.999984
2   15707 days 23:59:59.999984
dtype: timedelta64[ns]

In [6]: dti - np.datetime64('2013-01-01')
Out[6]: DatetimeIndex(['1970-01-01', '1970-01-02', '1970-01-03'], dtype='datetime64[ns]', freq=None)
Member

jbrockmendel commented Oct 30, 2017

Does this also fix #7996 ?

No, but it does fix a related bug that probably belongs in the same issue

In [3]: s = pd.Series(pd.date_range('20130101',periods=3))
In [4]: dti = pd.DatetimeIndex(s)

In [5]: s-np.datetime64('2013-01-01')
Out[5]: 
0   15705 days 23:59:59.999984
1   15706 days 23:59:59.999984
2   15707 days 23:59:59.999984
dtype: timedelta64[ns]

In [6]: dti - np.datetime64('2013-01-01')
Out[6]: DatetimeIndex(['1970-01-01', '1970-01-02', '1970-01-03'], dtype='datetime64[ns]', freq=None)
@jbrockmendel

This comment has been minimized.

Show comment
Hide comment
@jbrockmendel

jbrockmendel Oct 30, 2017

Member

Rebased and pushed; hoping that magically fixes the CI errors

Member

jbrockmendel commented Oct 30, 2017

Rebased and pushed; hoping that magically fixes the CI errors

@jbrockmendel

This comment has been minimized.

Show comment
Hide comment
@jbrockmendel

jbrockmendel Oct 30, 2017

Member

AFAICT the test failures here were caused because of fragility in test_NaT_methods and TestTimestamp.test_to_datetime_depr. Specifically, a test that this PR added (until moments ago) called ts.to_datetime() instead of ts.to_pydatetime(). This triggered a FutureWarning, which resulted in FutureWarning not being called in the tests that specifically check for it.

I think the immediate issue is now fixed, but ideally test_to_datetime_depr and test_NaT_methods would be isolated enough not to depend on test ordering.

Member

jbrockmendel commented Oct 30, 2017

AFAICT the test failures here were caused because of fragility in test_NaT_methods and TestTimestamp.test_to_datetime_depr. Specifically, a test that this PR added (until moments ago) called ts.to_datetime() instead of ts.to_pydatetime(). This triggered a FutureWarning, which resulted in FutureWarning not being called in the tests that specifically check for it.

I think the immediate issue is now fixed, but ideally test_to_datetime_depr and test_NaT_methods would be isolated enough not to depend on test ordering.

@jreback jreback added this to the 0.21.1 milestone Oct 31, 2017

@jreback

needs a whatsnew note. 0.21.1 is fine.

Show outdated Hide outdated pandas/tests/indexes/datetimes/test_datetime.py Outdated
Show outdated Hide outdated pandas/tests/indexes/timedeltas/test_ops.py Outdated
Show outdated Hide outdated pandas/tests/indexes/datetimes/test_datetime.py Outdated
@jbrockmendel

This comment has been minimized.

Show comment
Hide comment
@jbrockmendel

jbrockmendel Oct 31, 2017

Member

Where did we land on this? My preference is to get this bug-fix in and worry about the rest in #18049+followups.

Member

jbrockmendel commented Oct 31, 2017

Where did we land on this? My preference is to get this bug-fix in and worry about the rest in #18049+followups.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 1, 2017

Contributor

needs a rebase

Contributor

jreback commented Nov 1, 2017

needs a rebase

@jbrockmendel

This comment has been minimized.

Show comment
Hide comment
@jbrockmendel

jbrockmendel Nov 1, 2017

Member

Just rebased. The two new tests are unchanged, just moved to the appropriate locations in test_arithmetic.

Member

jbrockmendel commented Nov 1, 2017

Just rebased. The two new tests are unchanged, just moved to the appropriate locations in test_arithmetic.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 4, 2017

Contributor

looks fine, tiny doc change. ping on green.

Contributor

jreback commented Nov 4, 2017

looks fine, tiny doc change. ping on green.

Show outdated Hide outdated doc/source/whatsnew/v0.21.1.txt Outdated
Show outdated Hide outdated doc/source/whatsnew/v0.21.1.txt Outdated
@jreback

jreback approved these changes Nov 4, 2017

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 4, 2017

Contributor

ping on green.

Contributor

jreback commented Nov 4, 2017

ping on green.

@jbrockmendel

This comment has been minimized.

Show comment
Hide comment
@jbrockmendel
Member

jbrockmendel commented Nov 4, 2017

ping

@jreback jreback merged commit 8388a47 into pandas-dev:master Nov 4, 2017

3 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Nov 4, 2017

Contributor

thanks!

Contributor

jreback commented Nov 4, 2017

thanks!

@jbrockmendel jbrockmendel deleted the jbrockmendel:arith branch Dec 8, 2017

TomAugspurger added a commit to TomAugspurger/pandas that referenced this pull request Dec 11, 2017

TomAugspurger added a commit to TomAugspurger/pandas that referenced this pull request Dec 11, 2017

TomAugspurger added a commit that referenced this pull request Dec 12, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment