fillna() containing timezone aware datetime64 rounded #14872

Closed
brendene opened this Issue Dec 13, 2016 · 1 comment

Comments

Projects
None yet
2 participants

Code Sample, a copy-pastable example if possible

import pandas as pd
import datetime, pytz

#Non-timezone aware datetime
s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001) ] )
print(s)
0                          NaT
1                          NaT
2   2016-12-12 22:24:06.100001
dtype: datetime64[ns]

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001
1   2016-12-12 22:24:06.100001
2   2016-12-12 22:24:06.100001
dtype: datetime64[ns]


# Timezone aware datetime
s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001, tzinfo=pytz.utc) ] )
print(s)
0                                NaT
1                                NaT
2   2016-12-12 22:24:06.100001+00:00

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001024+00:00
1   2016-12-12 22:24:06.100001024+00:00
2   2016-12-12 22:24:06.100001024+00:00
dtype: datetime64[ns, UTC]

Problem description

This is similar to bug #6587 but difference is a timezone aware datetime. It looks like the dtype is being stripped and the value is converted to a float and back.

Expected Output

s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001, tzinfo=pytz.utc) ] )
print(s)
0                                NaT
1                                NaT
2   2016-12-12 22:24:06.100001+00:00

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001000+00:00
1   2016-12-12 22:24:06.100001000+00:00
2   2016-12-12 22:24:06.100001000+00:00
dtype: datetime64[ns, UTC]

Output of pd.show_versions()

pandas: 0.19.1 numpy: 1.11.2 scipy: 0.18.1 pytz: 2016.7
Contributor

jreback commented Dec 13, 2016

yep, these prob just need a is_datetime64tz_dtype in the same path (see #6587 for where to add).

PR's welcome!

jreback added this to the Next Major Release milestone Dec 13, 2016

jreback closed this in f3c5a42 Dec 18, 2016

@jreback jreback modified the milestone: 0.19.2, Next Major Release Dec 18, 2016

@ischurov ischurov added a commit to ischurov/pandas that referenced this issue Dec 19, 2016

@opensourceworkAR @ischurov opensourceworkAR + ischurov BUG: .fillna() for datetime64 with tz is passing thru floats
closes #14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes #14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed
7a5ce35

@jorisvandenbossche jorisvandenbossche added a commit to jorisvandenbossche/pandas that referenced this issue Dec 24, 2016

@opensourceworkAR @jorisvandenbossche opensourceworkAR + jorisvandenbossche BUG: .fillna() for datetime64 with tz is passing thru floats
closes #14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes #14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed

(cherry picked from commit f3c5a42)
bbb7686

@ShaharBental ShaharBental added a commit to ShaharBental/pandas that referenced this issue Dec 26, 2016

@opensourceworkAR @ShaharBental opensourceworkAR + ShaharBental BUG: .fillna() for datetime64 with tz is passing thru floats
closes #14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes #14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed
fe87f91
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment