DataFrame.to_hdf append feature does behave properly

[aphearin]

I create a DataFrame and save it to disk as an hdf5 file, as follows:
df.to_hdf(fname, 'table')
This behaves as expected. But when I try to append to the existing hdf5 file,
df.to_hdf(fname, 'table',append=True)
pandas returns an error:
ValueError: Can only append to Tables

[TomAugspurger]

The error is saying that only "Table" type HDF stores can be appended to. There are two types

• fixed
• table

The default is fixed, so you create an HDFStore as df.to_hdf(fname, key='table', format='fixed'). Try it with format='table').key` is the identifier within the store.

[jorisvandenbossche]

The error message could maybe a bit more clear for who does not know that there are two formats?

[aphearin]

Yes, the error message could definitely be more clear.

The docs also lack clarity. It would be good if this particular point were made explicit in the docs, and also that pandas.to_hdf does not support appending tables with different column names. It would only take a sentence or two in the docs to clarify this.

[jreback]

the first 2 sections are all about this. http://pandas.pydata.org/pandas-docs/version/0.15.1/io.html#hdf5-pytables

appending different columns names raises an exception.

always happy to take doc updates though

[aphearin]

Yes, that is the section of the docs I was reading and find unclear. In particular, I think the following line copied from the docs is misleading:
df_tl.to_hdf('store_tl.h5','table',append=True)

This was the exact API I used that originally led me to submit the Issue. What would be more helpful would be an explicit example of a non-trivial append, together with some comment that the columns must be identical. Is there such a comment in the Read/Write API sub-section that I am just not seeing?

[TomAugspurger]

Agreed, that is unclear. Would you want to submit a pull request to clarify it?

I'd say the read/write section should be moved below the introduction of the two types, since that only works on table format.

[aphearin]

After taking a quick look at the rst, I think I'd rather just make my suggestion here.

The docs currently read:
df_tl = DataFrame(dict(A=list(range(5)), B=list(range(5))))
df_tl.to_hdf('store_tl.h5','table',append=True)
read_hdf('store_tl.h5', 'table', where = ['index>2'])

I'd suggest something like:

df_tl = DataFrame(dict(A=list(range(5)), B=list(range(5))))
df_tl.to_hdf('store_tl.h5',key='table')
read_hdf('store_tl.h5', key='table', where = ['index>2'])

"To append additional data to an existing hdf table, set both path and format to 'table', and be sure that the table you are appending has the same columns as the existing table in the hdf file:

df_tl1 = DataFrame(dict(A=list(range(5)), B=list(range(5))))
df_tl2 = DataFrame(dict(A=list(range(5)), B=list(range(5))))
df_tl1.to_hdf('store_tl.h5',key='table', format='table',append=True)
df_tl2.to_hdf('store_tl.h5',key='table', format='table',append=True)
read_hdf('store_tl.h5', 'table')

[jreback]

@aphearin a pull -request would be helpful.

[code-o-swagger]

@aphearin @jreback @TomAugspurger Hi Guys! I have a different kind of problem to solve. So I have data in the range of millions of columns and thousands of rows. For example, I am currently working on a file of size 10M rows x 1k columns, having a size of 50GB. When I am trying to read in the whole data, it's giving an Out of memory error. I am doing some operations, in which I need column by column access. So I want to read 10M rows x ~100 cols at a time, and then perform my operations, which are in-place modifications. Finally, I want to append all these outputs horizontally, and produce a final csv file. It seems to me that the append function only appends the data vertically. So can you please help me with how can I use HDF5 and it's append functionality to get this done. Thank you.

[jorisvandenbossche]

@aphearin Please don't post such questions on unrelated github issues. You can better use StackOverflow or the mailing list for this.