Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: DatetimeIndex.unique shifts tz-aware dates #21737

Closed
danielwlogan opened this issue Jul 4, 2018 · 5 comments

Comments

Projects
None yet
5 participants
@danielwlogan
Copy link

commented Jul 4, 2018

Code Sample, a copy-pastable example if possible

import pandas as pd
start = pd.Timestamp('2018-7-3 2:00:00', tz='America/Los_Angeles')
end = start + pd.Timedelta('36H')
a = pd.DatetimeIndex(start=start, end=end, freq='12H')
a.normalize().unique()

Problem description

The normalize method correctly sets the DatetimeIndex to midnight but the unique method returns a new DatetimeIndex not set to midnight, rather it returns a DatetimeIndex time as if converted to UTC (07:00 in the example code) but still retains the correct timezone ('America/Los_Angeles' in the example code). The code returns:

DatetimeIndex(['2018-07-03 07:00:00-07:00', '2018-07-04 07:00:00-07:00'], dtype='datetime64[ns, America/Los_Angeles]', freq=None)

Expected Output

DatetimeIndex(['2018-07-03 00:00:00-07:00', '2018-07-04 00:00:00-07:00'], dtype='datetime64[ns, America/Los_Angeles]', freq=None)

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-24-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_CA.UTF-8
LOCALE: en_CA.UTF-8

pandas: 0.23.1
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: 0.28.3
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: 0.4.1
pandas_datareader: None

@jschendel

This comment has been minimized.

Copy link
Member

commented Jul 5, 2018

Thanks, I can confirm that this issue is occurring on master.

A simpler demonstration of the bug:

In [2]: pd.__version__
Out[2]: '0.24.0.dev0+219.g1070976'

In [3]: dti = pd.DatetimeIndex(['2017', '2017'], tz='US/Eastern')

In [4]: dti
Out[4]: DatetimeIndex(['2017-01-01 00:00:00-05:00', '2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [5]: dti.unique()
Out[5]: DatetimeIndex(['2017-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

Surprisingly, we actually have a test for this, but it doesn't look to be working as intended:

@pytest.mark.parametrize('arr, expected', [
(pd.DatetimeIndex(['2017', '2017']), pd.DatetimeIndex(['2017'])),
(pd.DatetimeIndex(['2017', '2017'], tz='US/Eastern'),
pd.DatetimeIndex(['2017'], tz='US/Eastern')),
])
def test_unique(self, arr, expected):
result = arr.unique()
tm.assert_index_equal(result, expected)

Manually inspecting the test shows that the comparing the indexes themselves returns True, but comparing the index values individuals returns False:

In [6]: result = dti.unique()

In [7]: result
Out[7]: DatetimeIndex(['2017-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [8]: expected = pd.DatetimeIndex(['2017'], tz='US/Eastern')

In [9]: expected
Out[9]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [10]: result == expected
Out[10]: array([ True])

In [11]: result[0] == expected[0]
Out[11]: False

Notice that result and expected have the same .values:

In [12]: result.values
Out[12]: array(['2017-01-01T05:00:00.000000000'], dtype='datetime64[ns]')

In [13]: expected.values
Out[13]: array(['2017-01-01T05:00:00.000000000'], dtype='datetime64[ns]')

This has something to do with what DatetimeIndex.unique is doing under the hood, since attempting to reconstruct this from the Timestamp objects in question fails to reproduce the comparisons above:

In [14]: result2 = pd.DatetimeIndex([result[0]])

In [15]: result2
Out[15]: DatetimeIndex(['2017-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [16]: expected2 = pd.DatetimeIndex([expected[0]])

In [17]: expected2
Out[17]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [18]: result2 == expected2
Out[18]: array([False])

In [19]: result2.values
Out[19]: array(['2017-01-01T10:00:00.000000000'], dtype='datetime64[ns]')

In [20]: expected2.values
Out[20]: array(['2017-01-01T05:00:00.000000000'], dtype='datetime64[ns]')

Finally, note that pd.unique appears to be working correctly:

In [21]: dti
Out[21]: DatetimeIndex(['2017-01-01 00:00:00-05:00', '2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [22]: pd.unique(dti)
Out[22]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

@jschendel jschendel changed the title Getting a Unique Normalized DatetimeIndex BUG: DatetimeIndex.unique shifts tz-aware dates Jul 5, 2018

@Setur-sjd

This comment has been minimized.

Copy link

commented Jul 6, 2018

I faced the same bug yesterday.
It took me very long to find the bug in the big code and funnily the bug does not occur in pandas 0.22.0.
I have pandas 0.23.0
And However something very strange happens that I reproduced and copy here (Sypder Ipython):

df.head(2)
Out[17]: 
timestamp
2018-01-01 00:00:00.509000+01:00    134240
2018-01-01 00:00:02.509000+01:00    134350
Name: Whatever, dtype: int64

df.tail(2)
Out[18]: 
timestamp
2018-04-17 18:37:08.973000+02:00    121490
2018-04-17 18:37:10.973000+02:00    121510
Name: Whatever, dtype: int64

all_days = df.index.normalize().unique()

all_days[0]
Out[20]: Timestamp('2017-12-31 23:00:00+0100', tz='Europe/Berlin')

all_days
Out[21]: 
DatetimeIndex(['2018-01-01 00:00:00+01:00', '2018-01-02 00:00:00+01:00',
               '2018-01-03 00:00:00+01:00', '2018-01-04 00:00:00+01:00',
               '2018-01-05 00:00:00+01:00', '2018-01-06 00:00:00+01:00',
               '2018-01-07 00:00:00+01:00', '2018-01-08 00:00:00+01:00',
               '2018-01-09 00:00:00+01:00', '2018-01-10 00:00:00+01:00',
               ...
               '2018-04-08 00:00:00+02:00', '2018-04-09 00:00:00+02:00',
               '2018-04-10 00:00:00+02:00', '2018-04-11 00:00:00+02:00',
               '2018-04-12 00:00:00+02:00', '2018-04-13 00:00:00+02:00',
               '2018-04-14 00:00:00+02:00', '2018-04-15 00:00:00+02:00',
               '2018-04-16 00:00:00+02:00', '2018-04-17 00:00:00+02:00'],
              dtype='datetime64[ns, Europe/Berlin]', name='timestamp', length=107, freq=None)

However, to reproduce it in Jupyter Notebook (just being curious about it), the bug is shown even different!!

all_days = df.index.normalize().unique()

And that is

all_day[0]

:

Timestamp('2017-12-31 00:00:00')

@danielwlogan @jschendel

@jreback jreback added this to the 0.23.3 milestone Jul 6, 2018

@mroeschke

This comment has been minimized.

Copy link
Member

commented Jul 6, 2018

So there's some funniness when passing a DatetimeIndex into DatetimeIndex._simple_new. This is essentially what happens:

# .unique() takes uniques on UTC dates
In [13]: pd.DatetimeIndex._simple_new(pd.DatetimeIndex(['2017-01-01 05:00:00']), tz='US/Eastern')
Out[13]: DatetimeIndex(['2017-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [14]: pd.DatetimeIndex._simple_new(pd.DatetimeIndex(['2017-01-01 05:00:00']).values, tz='US/Eastern')
Out[14]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

So there might be a bug that lies deeper when setting a DatetimeIndex as opposed to a numpy array on the Block.

But adding the .values makes this correct, as In [11]: in @jschendel's example then returns True

(pandas-dev) matthewroeschke:pandas-mroeschke matthewroeschke$ git diff
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
index 966eff582..42a2e531d 100644
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -1072,8 +1072,7 @@ class DatetimeIndex(DatetimeArrayMixin, DatelikeOps, TimelikeOps,
         else:
             naive = self
         result = super(DatetimeIndex, naive).unique(level=level)
-        return self._simple_new(result, name=self.name, tz=self.tz,
-                                freq=self.freq)
+        return self._simple_new(result.values, name=self.name, tz=self.tz, freq=self.freq)

     def union(self, other):
         """
# With .values patch

In [1]: dti = pd.DatetimeIndex(['2017', '2017'], tz='US/Eastern')

In [2]: result = dti.unique()

In [3]: expected = pd.DatetimeIndex(['2017'], tz='US/Eastern')

In [5]: result
Out[5]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [6]: expected
Out[6]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [7]: result == expected
Out[7]: array([ True])

In [8]: result[0] == expected[0]
Out[8]: True

@mroeschke

This comment has been minimized.

Copy link
Member

commented Jul 31, 2018

So this appears fixed on master: (maybe due to #20912)

In [1]: dti = pd.DatetimeIndex(['2017', '2017'], tz='US/Eastern')

In [2]: dti.unique()
Out[2]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [3]: result = dti.unique()

In [4]:  expected = pd.DatetimeIndex(['2017'], tz='US/Eastern')

In [5]: result == expected
Out[5]: array([ True])

In [6]: result[0] == expected[0]
Out[6]: True

In [7]: pd.__version__
Out[7]: '0.24.0.dev0+384.gc272c52a5'

Although we already have a test for unique it would be great to additionally test that the underlying timestamps are equal as noted by @jschendel

@jreback jreback modified the milestones: 0.23.4, 0.24.0 Aug 2, 2018

@mroeschke mroeschke referenced this issue Aug 5, 2018

Merged

CLN: Old timezone issues #22201

8 of 8 tasks complete
@danielwlogan

This comment has been minimized.

Copy link
Author

commented Aug 7, 2018

Thanks everyone for your work tracking this down and getting a fix in place quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.