ENH: Export DataFrame containing datetime with tz to HDF5 in fixed format #11411

Closed
Tux1 opened this Issue Oct 22, 2015 · 6 comments

Comments

Projects
None yet
2 participants
Contributor

Tux1 commented Oct 22, 2015

Hello,

I noticed a bug about exporting DataFrame with columns containing localized datetime. I might be due to some another required libraries updated...
Example :

a = pd.to_datetime(['20120201', '20130201'])
b = pd.to_datetime(['20120201', '20130201'])
dfa = pd.DataFrame([a])
dfb = pd.DataFrame([b])
dfb[0] = dfb[0].dt.tz_localize('UTC')

dfa.to_hdf('tata.hdf', 'df') # success
dfb.to_hdf('toto.hdf', 'df') # fail

It works fine with Series
Thanks guys

Contributor

jreback commented Oct 22, 2015

you need to use format='table', this is not implemented for format='fixed', the default.

jreback closed this Oct 22, 2015

Contributor

Tux1 commented Oct 22, 2015

However, it was working until pandas 0.17, no ?

Contributor

jreback commented Oct 22, 2015

It was never supported in any real way (and not tested), so it 'worked' I guess.

In [7]: dfb.to_hdf('toto.hdf', 'df') # fail
/Users/jreback/miniconda/envs/pandas_0.16.2/lib/python2.7/site-packages/pandas/io/pytables.py:2577: PerformanceWarning: 
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->datetime,key->block1_values] [items->[0]]

  warnings.warn(ws, PerformanceWarning)
Contributor

jreback commented Oct 22, 2015

this is a dupe of #8165

if you'd like to work on that would be great.

Contributor

Tux1 commented Oct 22, 2015

Not very easy but I will try yes.
Thanks

Contributor

jreback commented Oct 22, 2015

actually, will reopen this one and close the other

jreback reopened this Oct 22, 2015

jreback added this to the Next Major Release milestone Oct 22, 2015

jreback changed the title from Export DataFrame containing datetime with tz doesn't work in 0.17 to ENH: Export DataFrame containing datetime with tz to HDF5 in fixed format Oct 22, 2015

@jreback jreback modified the milestone: 0.17.1, Next Major Release Nov 17, 2015

@jreback jreback added a commit to jreback/pandas that referenced this issue Nov 17, 2015

@jreback jreback ENH: Implement export of datetime64[ns, tz] dtypes with a fixed HDF5 …
…store #11411
addcec2

jreback closed this in #11628 Nov 17, 2015

@jreback jreback added a commit that referenced this issue Nov 17, 2015

@jreback jreback Merge pull request #11628 from jreback/hdf
ENH: Implement export of datetime64[ns, tz] dtypes with a fixed HDF5 #11411
b54f2db
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment