Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DatetimeIndex.union with tz-aware and tz-naive raises TypeError #21671

Closed
mroeschke opened this issue Jun 29, 2018 · 5 comments

Comments

@mroeschke
Copy link
Member

commented Jun 29, 2018

xref #11351

In [28]: s = pd.Series([pd.Timestamp('2011-01-01'), pd.NaT])

In [29]: stz = pd.Series(pd.DatetimeIndex(['2012-01-01', '2012-01-02'], tz='Asia/Tokyo'))

In [30]: pd.Index(s).union(pd.Index(stz))

TypeError: Cannot join tz-naive with tz-aware DatetimeIndex

In [32]: pd.__version__
Out[32]: '0.24.0.dev0+192.g0b63e81.dirty'

Problem description

Per #11351 (comment), this operation should just convert the data to object and not raise a TypeError.

Expected

In [33]: pd.Index(s).append(pd.Index(stz))
Out[33]:
Index([2011-01-01 00:00:00, NaT, 2012-01-01 00:00:00+09:00,
       2012-01-02 00:00:00+09:00],
      dtype='object')
@gfyoung

This comment has been minimized.

Copy link
Member

commented Jul 4, 2018

Don't forget though that the union will do object comparison at the element-level, where that timezone incompatibility still exists so "casting to object" will issue warnings like this:

Traceback (most recent call last):
  File "pandas/_libs/tslibs/timestamps.pyx", line 171, in pandas._libs.tslibs.timestamps._Timestamp.__richcmp__
  File "pandas/_libs/tslibs/timestamps.pyx", line 231, in pandas._libs.tslibs.timestamps._Timestamp._assert_tzawareness_compat
TypeError: Cannot compare tz-naive and tz-aware timestamps
Exception ignored in: 'pandas._libs.lib.array_equivalent_object'
Traceback (most recent call last):
  File "pandas/_libs/tslibs/timestamps.pyx", line 171, in pandas._libs.tslibs.timestamps._Timestamp.__richcmp__
  File "pandas/_libs/tslibs/timestamps.pyx", line 231, in pandas._libs.tslibs.timestamps._Timestamp._assert_tzawareness_compat
TypeError: Cannot compare tz-naive and tz-aware timestamps
sys:1: RuntimeWarning: Cannot compare tz-naive and tz-aware timestamps, sort order is undefined for incomparable objects
Index([2011-01-01 00:00:00, NaT, 2012-01-01 00:00:00+09:00,
       2012-01-02 00:00:00+09:00],
      dtype='object')

Do we just want to swallow those or suppress them? I don't want to overhaul __richcmp__ just for this (corner) case admittedly.

Here is the current attempt:

forking-repos@392bd53

@gfyoung

This comment has been minimized.

Copy link
Member

commented Jul 4, 2018

Actually, are we even able to redirect the Cython stderr without doing some kind of dance?

See this SO post here.

@gfyoung

This comment has been minimized.

Copy link
Member

commented Jul 5, 2018

@jreback : Thoughts?

@jbrockmendel

This comment has been minimized.

Copy link
Member

commented Jun 18, 2019

@mroeschke this appears to work in master. closeable?

@mroeschke

This comment has been minimized.

Copy link
Member Author

commented Jun 18, 2019

Probably could use a unit test. I don't think we have one for this use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.