New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fillna() containing timezone aware datetime64 rounded #14872

Closed
brendene opened this Issue Dec 13, 2016 · 1 comment

Comments

Projects
None yet
2 participants
@brendene

Code Sample, a copy-pastable example if possible

import pandas as pd
import datetime, pytz

#Non-timezone aware datetime
s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001) ] )
print(s)
0                          NaT
1                          NaT
2   2016-12-12 22:24:06.100001
dtype: datetime64[ns]

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001
1   2016-12-12 22:24:06.100001
2   2016-12-12 22:24:06.100001
dtype: datetime64[ns]


# Timezone aware datetime
s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001, tzinfo=pytz.utc) ] )
print(s)
0                                NaT
1                                NaT
2   2016-12-12 22:24:06.100001+00:00

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001024+00:00
1   2016-12-12 22:24:06.100001024+00:00
2   2016-12-12 22:24:06.100001024+00:00
dtype: datetime64[ns, UTC]

Problem description

This is similar to bug #6587 but difference is a timezone aware datetime. It looks like the dtype is being stripped and the value is converted to a float and back.

Expected Output

s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001, tzinfo=pytz.utc) ] )
print(s)
0                                NaT
1                                NaT
2   2016-12-12 22:24:06.100001+00:00

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001000+00:00
1   2016-12-12 22:24:06.100001000+00:00
2   2016-12-12 22:24:06.100001000+00:00
dtype: datetime64[ns, UTC]

Output of pd.show_versions()

pandas: 0.19.1 numpy: 1.11.2 scipy: 0.18.1 pytz: 2016.7
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Dec 13, 2016

Contributor

yep, these prob just need a is_datetime64tz_dtype in the same path (see #6587 for where to add).

PR's welcome!

Contributor

jreback commented Dec 13, 2016

yep, these prob just need a is_datetime64tz_dtype in the same path (see #6587 for where to add).

PR's welcome!

@jreback jreback added this to the Next Major Release milestone Dec 13, 2016

@jreback jreback closed this in f3c5a42 Dec 18, 2016

@jreback jreback modified the milestones: 0.19.2, Next Major Release Dec 18, 2016

ischurov added a commit to ischurov/pandas that referenced this issue Dec 19, 2016

BUG: .fillna() for datetime64 with tz is passing thru floats
closes #14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes #14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed

jorisvandenbossche added a commit to jorisvandenbossche/pandas that referenced this issue Dec 24, 2016

BUG: .fillna() for datetime64 with tz is passing thru floats
closes #14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes #14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed

(cherry picked from commit f3c5a42)

ShaharBental added a commit to ShaharBental/pandas that referenced this issue Dec 26, 2016

BUG: .fillna() for datetime64 with tz is passing thru floats
closes #14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes #14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment