Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: 0.12.0 using DataFrame.to_hdf() with mode='a' does not append the data #4584

Closed
bluefir opened this issue Aug 16, 2013 · 5 comments · Fixed by #4645

Comments

@bluefir
Copy link

commented Aug 16, 2013

to_hdf(....., append=True,table=True) raising, should accept this

Is there a way to use DataFrame.to_hdf() so that data can be appened the data? Somehow to_hdf(filename, table_name, mode='a', table=True) overwrites the data while to_hdf(filename, table_name, table=True, append=True) gives the following error:

Traceback (most recent call last):
File "M:/Projects/PortfolioAnalytics/Code/Python/attribution/TopContributors.py", line 1035, in
portfolio_data.to_hdf(filename, OUTPUT_NAME_PORTFOLIO, table=True, append=Settings.TC_OUTPUT_APPEND)
File "C:\Python27\lib\site-packages\pandas\core\generic.py", line 521, in to_hdf
return pytables.to_hdf(path_or_buf, key, self, *_kwargs)
File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 194, in to_hdf
f(store)
File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 188, in
f = lambda store: store.append(key, value, *_kwargs)
File "C:\Python27\lib\site-packages\pandas\io\pytables.py", line 658, in append
self._write_to_group(key, value, table=True, append=True, **kwargs)
TypeError: _write_to_group() got multiple values for keyword argument 'table'

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2013

table=True implies append (so don't need append keyword (its really a relic anyhow)), and this is a PER GROUP option

mode has to do with the file, e.g. passing mode=w overwrites the file, normally you should ALWAYS use the default mode=a (as you can read/write with it)

@bluefir

This comment has been minimized.

Copy link
Author

commented Aug 16, 2013

Well, using

df1.to_hdf(filename, table_name, mode='w', table=True)

and then

df2.to_hdf(filename, table_name, mode='a', table=True)

overwrites df1 data in table_name with df2 data.

pd.read_hdf(filename, table_name)

after df1

<class 'pandas.core.frame.DataFrame'>
MultiIndex: 108879 entries, (2013-08-01 00:00:00, 000312) to (2013-08-09 00:00:00, Y8565N10)
Data columns (total 12 columns):
market_cap 108830 non-null values
alpha 108676 non-null values
gics_code 108830 non-null values
investable 108879 non-null values
issuer_country 108701 non-null values
msci_country 77291 non-null values
universe 108879 non-null values
weight_benchmark 108834 non-null values
weight_portfolio 108834 non-null values
weight_active 108834 non-null values
country 108834 non-null values
return_daily 108649 non-null values
dtypes: bool(2), float64(7), object(3)

after df2

<class 'pandas.core.frame.DataFrame'>
MultiIndex: 62432 entries, (2013-08-12 00:00:00, 000312) to (2013-08-15 00:00:00, Y8565N10)
Data columns (total 12 columns):
market_cap 62432 non-null values
alpha 62344 non-null values
gics_code 62432 non-null values
investable 62432 non-null values
issuer_country 62356 non-null values
msci_country 44268 non-null values
universe 62432 non-null values
weight_benchmark 62432 non-null values
weight_portfolio 62432 non-null values
weight_active 62432 non-null values
country 62432 non-null values
return_daily 62281 non-null values
dtypes: bool(2), float64(7), object(3)

Am I missing something?

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2013

try to 2nd one with append=True and leave table=True out. append=True implies table, so you only need one of these.

I think append=True,table=True is a bug (in that it should work and not cause an error)

FYI....prob going to deprecate the table options and instead

format='storer | `table(default will bestorer``)

and append option will still be there

@bluefir

This comment has been minimized.

Copy link
Author

commented Aug 16, 2013

Thanks!

@bluefir bluefir closed this Aug 16, 2013

@jreback

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2013

oh....this is a 'bug'...going to fix..thanks for the report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.