Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fillna() containing timezone aware datetime64 rounded #14872

Closed
brendene opened this issue Dec 13, 2016 · 1 comment
Closed

fillna() containing timezone aware datetime64 rounded #14872

brendene opened this issue Dec 13, 2016 · 1 comment
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timeseries
Milestone

Comments

@brendene
Copy link

Code Sample, a copy-pastable example if possible

import pandas as pd
import datetime, pytz

#Non-timezone aware datetime
s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001) ] )
print(s)
0                          NaT
1                          NaT
2   2016-12-12 22:24:06.100001
dtype: datetime64[ns]

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001
1   2016-12-12 22:24:06.100001
2   2016-12-12 22:24:06.100001
dtype: datetime64[ns]


# Timezone aware datetime
s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001, tzinfo=pytz.utc) ] )
print(s)
0                                NaT
1                                NaT
2   2016-12-12 22:24:06.100001+00:00

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001024+00:00
1   2016-12-12 22:24:06.100001024+00:00
2   2016-12-12 22:24:06.100001024+00:00
dtype: datetime64[ns, UTC]

Problem description

This is similar to bug #6587 but difference is a timezone aware datetime. It looks like the dtype is being stripped and the value is converted to a float and back.

Expected Output

s = pd.Series( [pd.NaT, pd.NaT, datetime.datetime(2016, 12, 12, 22, 24, 6, 100001, tzinfo=pytz.utc) ] )
print(s)
0                                NaT
1                                NaT
2   2016-12-12 22:24:06.100001+00:00

print(s.fillna(method='bfill'))
0   2016-12-12 22:24:06.100001000+00:00
1   2016-12-12 22:24:06.100001000+00:00
2   2016-12-12 22:24:06.100001000+00:00
dtype: datetime64[ns, UTC]

Output of pd.show_versions()

pandas: 0.19.1 numpy: 1.11.2 scipy: 0.18.1 pytz: 2016.7
@jreback
Copy link
Contributor

jreback commented Dec 13, 2016

yep, these prob just need a is_datetime64tz_dtype in the same path (see #6587 for where to add).

PR's welcome!

@jreback jreback added Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timeseries Difficulty Novice labels Dec 13, 2016
@jreback jreback added this to the Next Major Release milestone Dec 13, 2016
@jreback jreback modified the milestones: 0.19.2, Next Major Release Dec 18, 2016
ischurov pushed a commit to ischurov/pandas that referenced this issue Dec 19, 2016
closes pandas-dev#14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes pandas-dev#14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed
jorisvandenbossche pushed a commit to jorisvandenbossche/pandas that referenced this issue Dec 24, 2016
closes pandas-dev#14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes pandas-dev#14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed

(cherry picked from commit f3c5a42)
ShaharBental pushed a commit to ShaharBental/pandas that referenced this issue Dec 26, 2016
closes pandas-dev#14872

Author: Rodolfo Fernandez <opensourceworkAR@users.noreply.github.com>

Closes pandas-dev#14905 from RodolfoRFR/pandas-14872-e and squashes the following commits:

18802b4 [Rodolfo Fernandez] added 'self' to test_dtype_utc function in pandas/tests/series/test_missing
e0c6c7c [Rodolfo Fernandez] added line to whatsnew v0.19.2 and test to test_missing.py in series folder
e4ba7e0 [Rodolfo Fernandez] removed all references to _DATELIKE_DTYPES from /pandas/core/missing.py
5d37ce8 [Rodolfo Fernandez] added is_datetime64tz_dtype and changed evaluation from 'values' to dtype
19eecb2 [Rodolfo Fernandez] fixed style errors using flake8
59b91a1 [Rodolfo Fernandez] test modified
5a59eac [Rodolfo Fernandez] test modified
bc68bf7 [Rodolfo Fernandez] test modified
ba83fc8 [Rodolfo Fernandez] test
b7358de [Rodolfo Fernandez] bug fixed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Timeseries
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants