New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Export DataFrame containing datetime with tz to HDF5 in fixed format #11411

Closed
Tux1 opened this Issue Oct 22, 2015 · 6 comments

Comments

Projects
None yet
2 participants
@Tux1
Contributor

Tux1 commented Oct 22, 2015

Hello,

I noticed a bug about exporting DataFrame with columns containing localized datetime. I might be due to some another required libraries updated...
Example :

a = pd.to_datetime(['20120201', '20130201'])
b = pd.to_datetime(['20120201', '20130201'])
dfa = pd.DataFrame([a])
dfb = pd.DataFrame([b])
dfb[0] = dfb[0].dt.tz_localize('UTC')

dfa.to_hdf('tata.hdf', 'df') # success
dfb.to_hdf('toto.hdf', 'df') # fail

It works fine with Series
Thanks guys

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 22, 2015

Contributor

you need to use format='table', this is not implemented for format='fixed', the default.

Contributor

jreback commented Oct 22, 2015

you need to use format='table', this is not implemented for format='fixed', the default.

@Tux1

This comment has been minimized.

Show comment
Hide comment
@Tux1

Tux1 Oct 22, 2015

Contributor

However, it was working until pandas 0.17, no ?

Contributor

Tux1 commented Oct 22, 2015

However, it was working until pandas 0.17, no ?

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 22, 2015

Contributor

It was never supported in any real way (and not tested), so it 'worked' I guess.

In [7]: dfb.to_hdf('toto.hdf', 'df') # fail
/Users/jreback/miniconda/envs/pandas_0.16.2/lib/python2.7/site-packages/pandas/io/pytables.py:2577: PerformanceWarning: 
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->datetime,key->block1_values] [items->[0]]

  warnings.warn(ws, PerformanceWarning)
Contributor

jreback commented Oct 22, 2015

It was never supported in any real way (and not tested), so it 'worked' I guess.

In [7]: dfb.to_hdf('toto.hdf', 'df') # fail
/Users/jreback/miniconda/envs/pandas_0.16.2/lib/python2.7/site-packages/pandas/io/pytables.py:2577: PerformanceWarning: 
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->datetime,key->block1_values] [items->[0]]

  warnings.warn(ws, PerformanceWarning)
@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 22, 2015

Contributor

this is a dupe of #8165

if you'd like to work on that would be great.

Contributor

jreback commented Oct 22, 2015

this is a dupe of #8165

if you'd like to work on that would be great.

@Tux1

This comment has been minimized.

Show comment
Hide comment
@Tux1

Tux1 Oct 22, 2015

Contributor

Not very easy but I will try yes.
Thanks

Contributor

Tux1 commented Oct 22, 2015

Not very easy but I will try yes.
Thanks

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 22, 2015

Contributor

actually, will reopen this one and close the other

Contributor

jreback commented Oct 22, 2015

actually, will reopen this one and close the other

@jreback jreback reopened this Oct 22, 2015

@jreback jreback added this to the Next Major Release milestone Oct 22, 2015

@jreback jreback changed the title from Export DataFrame containing datetime with tz doesn't work in 0.17 to ENH: Export DataFrame containing datetime with tz to HDF5 in fixed format Oct 22, 2015

@jreback jreback modified the milestones: 0.17.1, Next Major Release Nov 17, 2015

jreback added a commit to jreback/pandas that referenced this issue Nov 17, 2015

jreback added a commit that referenced this issue Nov 17, 2015

Merge pull request #11628 from jreback/hdf
ENH: Implement export of datetime64[ns, tz] dtypes with a fixed HDF5 #11411
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment