Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

to_datetime uses previous row's timezone when timezone not specified and utc=True #24992

Closed
billmccord opened this issue Jan 29, 2019 · 2 comments

Comments

@billmccord
Copy link

commented Jan 29, 2019

Code Sample, a copy-pastable example if possible

Without utc=True:

pd.to_datetime([
    '2018-11-28T00:00:00', 
    '2018-11-28T00:00:00+12:00', 
    '2018-11-28T00:00:00', 
    '2018-11-28T00:00:00+06:00', 
    '2018-11-28T00:00:00'
])

Result:

Index([      2018-11-28 00:00:00, 2018-11-28 00:00:00+12:00,
             2018-11-28 00:00:00, 2018-11-28 00:00:00+06:00,
             2018-11-28 00:00:00],
      dtype='object')

With utc=True:

pd.to_datetime([
    '2018-11-28T00:00:00', 
    '2018-11-28T00:00:00+12:00', 
    '2018-11-28T00:00:00', 
    '2018-11-28T00:00:00+06:00', 
    '2018-11-28T00:00:00'
], utc=True)

Result:

DatetimeIndex(['2018-11-28 00:00:00+00:00', '2018-11-27 12:00:00+00:00',
               '2018-11-27 12:00:00+00:00', '2018-11-27 18:00:00+00:00',
               '2018-11-27 18:00:00+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)

Problem description

The behavior leads to unexpected and inconsistent treatment of datetimes where the timezone is not explicitly specified. Simply using the last timezone it saw from a datetime where the timezone was specified is pretty confusing.

Expected Output

I would be fine with it just assuming that any datetimes where the timezone is NOT specified are already in UTC. Another valid way of handling it would be to use os.environ['TZ']. Another valid way would be to add another input argument to to_datetime that allowed specifying which timezone to use when there is no timezone specified. Any of these would lead to conclusive, explainable results. This is what I would like to see:

DatetimeIndex(['2018-11-28 00:00:00+00:00', '2018-11-27 12:00:00+00:00',
               '2018-11-28 00:00:00+00:00', '2018-11-27 18:00:00+00:00',
               '2018-11-28 00:00:00+00:00'],
              dtype='datetime64[ns, UTC]', freq=None)

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.2.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0
pytest: None
pip: 19.0.1
setuptools: 40.7.0
Cython: None
numpy: 1.16.0
scipy: 1.2.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.9
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml.etree: 4.3.0
bs4: None
html5lib: None
sqlalchemy: 1.2.17
pymysql: None
psycopg2: 2.7.7 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

@TomAugspurger

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2019

cc @jbrockmendel @mroeschke if you have thoughts on the expected. This should probably be all True (order shouldn't matter).

In [21]: items = ['2018-11-28T00:00:00+12:00', '2018-11-28T00:00:00']

In [22]: a = pd.to_datetime(items, utc=True)

In [23]: b = pd.to_datetime(list(reversed(items)), utc=True)[::-1]

In [24]: a == b
Out[24]: array([ True, False])

The 0.24.0 behavior is the same as 0.23.4.

@jbrockmendel

This comment has been minimized.

Copy link
Member

commented Jan 29, 2019

Agreed on "order shouldn't matter". Something similar came up in #24006 with how we treat "now" and "today".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.