BUG/API: Make consistent API for tz-aware and tz-naive ops

[sinhrks]

Currently there are some inconsistent results. We should clarify the policy first.

import pandas as pd

s = pd.Series([pd.Timestamp('2011-01-01'), pd.NaT])
s
#0   2011-01-01
#1          NaT
# dtype: datetime64[ns]

1. coerced to object dtype

I prefer this behavior.

s.fillna(pd.Timestamp('2011-01-02', tz='Asia/Tokyo'))
#0          2011-01-01 00:00:00
#1    2011-01-02 00:00:00+09:00
# dtype: object

2. converted to GMT

s[1] = pd.Timestamp('2011-01-02', tz='Asia/Tokyo')
s
#0   2011-01-01 00:00:00
#1   2011-01-01 15:00:00
# dtype: datetime64[ns]

3. raise explicit error

This looks second best.

stz = pd.Series(pd.DatetimeIndex(['2012-01-01', '2012-01-02'], tz='Asia/Tokyo'))
stz
#0   2012-01-01 00:00:00+09:00
#1   2012-01-02 00:00:00+09:00
# dtype: datetime64[ns, Asia/Tokyo]

pd.Index(s).union(pd.Index(stz))
# TypeError: Cannot join tz-naive with tz-aware DatetimeIndex

4. not handled properly (bug)

s.append(stz)
# AttributeError: 'numpy.ndarray' object has no attribute 'tz_localize'

-> Split to #11455, closed by #12195.

[jorisvandenbossche]

Note that also with other dtypes there are some of the same 'inconsistencies', casting when assigning and not with fillna:

In [36]: s = pd.Series([1.4, np.nan])

In [37]: s.fillna('3')
Out[37]:
0    1.4
1      3
dtype: object

In [38]: s[1] = '3'

In [39]: s
Out[39]:
0    1.4
1    3.0
dtype: float64

[jreback]

@jorisvandenbossche your last example is numpy conversion at work, we could fix it, but IMHO nor really worth it.

Option 1 is already selected, and reaffirmed in #11329

So the general rule for fillna/replace is that downcasting happens when possible where no loss of information occurs:

In [3]: s = pd.Series([pd.Timestamp('2011-01-01'), pd.NaT],dtype='object')

In [4]: s
Out[4]: 
0    2011-01-01 00:00:00
1                    NaT
dtype: object

In [5]: s.fillna(Timestamp('20130101'))
Out[5]: 
0   2011-01-01
1   2013-01-01
dtype: datetime64[ns]

All other cases are converted to object, e.g. when using mixed timezones for example (or naive and a tz), or mixed numbers/strings.

I will leave this issue open to discuss a possibility of using Option 3 on a specific case (explicity raising an error) (or having a raise_on_error flag to fillna/replace or somesuch.

using .fillna with a naive tz with a tz based Series (or vice-versa) should really just raise the error and not convert to object as this is almost certainly a user error, though its ok as we have it now.

[4] above

In [1]:  s = pd.Series([pd.Timestamp('2011-01-01',tz='US/Eastern'), pd.NaT])

In [2]: s
Out[2]: 
0   2011-01-01 00:00:00-05:00
1                         NaT
dtype: datetime64[ns, US/Eastern]

In [3]: s.fillna(Timestamp('20130101'))
Out[3]: 
0    2011-01-01 00:00:00-05:00
1          2013-01-01 00:00:00
dtype: object

In [4]: s.fillna(Timestamp('20130101',tz='US/Pacific'))
Out[4]: 
0    2011-01-01 00:00:00-05:00
1    2013-01-01 00:00:00-08:00
dtype: object

[jreback]

@sinhrks

[3] - we actually allow this and just convert things to object dtype, so that is a bug as well (again maybe we should have an option to these, then change the default to raise)
[4] is a bug, pls open a separate issue for that.

[jreback]

cc @mroeschke some of these might be closable as already fixed (future PR)

[mroeschke]

Everything looks tested and fixed but [3], opened up #21671 for a more specific issue.