New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: DatetimeIndex.time incorrect in case of timezones #21267

Closed
ssikdar1 opened this Issue May 31, 2018 · 11 comments

Comments

Projects
None yet
4 participants
@ssikdar1
Contributor

ssikdar1 commented May 31, 2018

An offshoot of #21230

http://pandas.pydata.org/pandas-docs/stable/timeseries.html?highlight=weekofmonth#time-date-components

From the docs:
There are several time/date properties that one can access from Timestamp or a collection of timestamps like a DatetimeIndex.
...
time | Returns datetime.time (does not contain timezone information)

But take this code here:

import pandas as pd

df1 = pd.DataFrame(data=[24, 25],
                   index=pd.DatetimeIndex(['2013-01-24 15:01:00+01:00',
                                           '2013-01-25 15:01:00+01:00'],
                                          dtype='datetime64[ns, CET]',
                                          name='Date', freq=None))

print(type(df1.index))
print("df1.index.time")
print(df1.index.time)
                 

returns:

<class 'pandas.core.indexes.datetimes.DatetimeIndex'>
df1.index.time
[datetime.time(14, 1, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)
 datetime.time(14, 1, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)]


  • This is showing timezone information.

This may not be a bad thing, but if tz information is expected the docs should probably be changed?

[paste the output of pd.show_versions() here below this line]

pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-43-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: None
pip: 9.0.1
setuptools: 28.8.0
Cython: None
numpy: 1.14.3
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@mroeschke

This comment has been minimized.

Member

mroeschke commented May 31, 2018

Agreed, the docs should be changed to indicate that .time results may hold timezone information. Additionally we should ensure that tests exist for this behavior (didn't find any after a quick skim). PR's welcome!

@uds5501

This comment has been minimized.

Contributor

uds5501 commented May 31, 2018

@ssikdar1 @mroeschke I am sorry but I couldn't recreate the output that your code showed. Is it possible that it is an OS based feature? (mine is Windows)
image

Any other method by which that output could be achieved?

@mroeschke

This comment has been minimized.

Member

mroeschke commented May 31, 2018

What version of pandas are you using @uds5501? I don't believe this should be OS specific, and I can confirm this on the latest release.

In [11]: pd.__version__
Out[11]: u'0.23.0'

In [12]: d = pd.DatetimeIndex(['2013-01-24 14:01:00+01:00'], dtype='datetime64[ns, CET]', name='Date')

In [20]: d.time
Out[20]:
array([datetime.time(14, 1, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)],
      dtype=object)
@ssikdar1

This comment has been minimized.

Contributor

ssikdar1 commented May 31, 2018

So it works for pandas 0.22.0, is it possible then its a bug from 22->23?

Python 3.6.5 (default, Apr 25 2018, 14:23:58) 
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> df1 = pd.DataFrame(data=[24, 25],
...                    index=pd.DatetimeIndex(['2013-01-24 15:01:00+01:00',
...                                            '2013-01-25 15:01:00+01:00'],
...                                           dtype='datetime64[ns, CET]',
...                                           name='Date', freq=None))
>>> 
>>> print(type(df1.index))
<class 'pandas.core.indexes.datetimes.DatetimeIndex'>
>>> print("df1.index.time")
df1.index.time
>>> print(df1.index.time)
[datetime.time(14, 1) datetime.time(14, 1)]
>>> 
>>> 
>>> 
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Darwin
OS-release: 17.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0

@mroeschke

This comment has been minimized.

Member

mroeschke commented May 31, 2018

So it appears this behavior changed after #18461. However IMO, I would consider this an coincidental enhancement instead of a regression (the time is still correct and now gained timezone information).

Thoughts @jorisvandenbossche?

@jorisvandenbossche

This comment has been minimized.

Member

jorisvandenbossche commented May 31, 2018

I actuallly get a different output as you: with a timezone (so changed from 0.22), but also the wrong hour (similar actually as in the top post):

In [50]: d = pd.DatetimeIndex(['2013-01-24 14:01:00+01:00'], dtype='datetime64[ns, CET]', name='Date')

In [51]: d.time
Out[51]: array([datetime.time(13, 1, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)], dtype=object)

so this is clearly a regression.

The fact that it is working correctly for you and not for me is strange .. What is the locale on your system? (in case it depends on that)

@mroeschke

This comment has been minimized.

Member

mroeschke commented May 31, 2018

Oops sorry. You're absolutely correct @jorisvandenbossche. Not sure what I was looking at but I am getting the same results as you now.

@jorisvandenbossche

This comment has been minimized.

Member

jorisvandenbossche commented May 31, 2018

It's from a similar PR #18461 as the related issue #21230 (which was from PR #18163)

@uds5501

This comment has been minimized.

Contributor

uds5501 commented Jun 1, 2018

@mroeschke I am using '0.21.1'

@mroeschke

This comment has been minimized.

Member

mroeschke commented Jun 2, 2018

@jorisvandenbossche so this is actually correct

In [2]: pd.__version__
Out[2]: u'0.23.0'

In [3]: d = pd.DatetimeIndex(['2013-01-24 14:01:00+01:00'], dtype='datetime64[ns, CET]', name='Date')

In [4]: d
Out[4]: DatetimeIndex(['2013-01-24 13:01:00+01:00'], dtype='datetime64[ns, CET]', name='Date', freq=None)

In [5]: d.time
Out[5]:
array([datetime.time(13, 1, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)],
      dtype=object)

But this confusion stems from this construction bug in #15938 (I believe)

@jorisvandenbossche jorisvandenbossche changed the title from timeseries documentation DatetimeIndex inconsistency? to BUG: DatetimeIndex.time incorrect in case of timezones Jun 4, 2018

@jorisvandenbossche

This comment has been minimized.

Member

jorisvandenbossche commented Jun 5, 2018

@jorisvandenbossche so this is actually correct

Ah, yes, sorry, missed this comment. It's indeed the construction that is confusing here!

jorisvandenbossche added a commit that referenced this issue Jun 7, 2018

BUG: Using DatetimeIndex.date with timezone returns incorrect date (#…
…21281)

* BUG: Using DatetimeIndex.date with timezone returns incorrect date #21230
* Fix bug where DTI.time returns a tz-aware Time instead of tz-naive #21267

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Jun 12, 2018

BUG: Using DatetimeIndex.date with timezone returns incorrect date (p…
…andas-dev#21281)

* BUG: Using DatetimeIndex.date with timezone returns incorrect date pandas-dev#21230
* Fix bug where DTI.time returns a tz-aware Time instead of tz-naive pandas-dev#21267

(cherry picked from commit a363e1a)

TomAugspurger added a commit that referenced this issue Jun 12, 2018

BUG: Using DatetimeIndex.date with timezone returns incorrect date (#…
…21281)

* BUG: Using DatetimeIndex.date with timezone returns incorrect date #21230
* Fix bug where DTI.time returns a tz-aware Time instead of tz-naive #21267

(cherry picked from commit a363e1a)

david-liu-brattle-1 added a commit to david-liu-brattle-1/pandas that referenced this issue Jun 18, 2018

BUG: Using DatetimeIndex.date with timezone returns incorrect date (p…
…andas-dev#21281)

* BUG: Using DatetimeIndex.date with timezone returns incorrect date pandas-dev#21230
* Fix bug where DTI.time returns a tz-aware Time instead of tz-naive pandas-dev#21267
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment