DataFrame Memory Usage after HDF5 retrieve

[CarstVaartjes]

Hi,

Pandas seems to be using way too much memory after retrieving data from a HDF5 store. In general memory peaks while reading (up to 30 times of the final data step) and then (even after calling the garbage collector) it uses around 10x the memory.
If I then get the numpy arrays out, delete the dataframe and create a new one, the new dataframe is memory efficient and what you would expect.
Please find an example underneath; the memory usage is what I see in "top" being used by ipython.

Create test data first:

#Create Testdata
import pandas as pd
import numpy as np
test_df = pd.DataFrame(np.random.randn(5000000, 26), columns=['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'])
test_df[['a', 'b']] = test_df[['a', 'b']].astype(int)

with pd.get_store(path='test.h5', mode='w', complib='blosc', complevel=9) as store_hdf:
    store_hdf.append('store_0', test_df, data_columns=['a', 'b'])

Now leave (i)python and go back into it and run the test:

import pandas as pd
import numpy as np
import gc
#44MB

path_name = '/srv/www/li/test.h5'
with pd.get_store(path_name, mode='r') as fact_hdf:
    test_df = fact_hdf.select('store_0', columns=['a', 'c'])

# peaks at 3.6GB, ends at 3.1GB (drops to 2.1GB a little later)

print test_df.values.nbytes / (1024*1024)
# 76MB
print test_df.index.nbytes / (1024*1024)
# 38MB

gc.collect()
#1.2GB

a = test_df['a'].values
c = test_df['c'].values
# No impact on memory usage

del test_df
gc.collect()
# drops to 154MB

test_df = pd.DataFrame({'a': a, 'c': c})
# total memory usage goes up to 268MB
print test_df.values.nbytes / (1024*1024)
# 76MB
print test_df.index.nbytes / (1024*1024)
# 38MB

# now delete the numpy arrays
del a
del c
gc.collect()
# 229MB

del test_df
gc.collect()
# 115MB

Machine Info:

• Ubuntu 12.04LTS 64bit
• Pandas 0.13.1
• Pytables 3.1.0

Any ideas on what is happening?

[jreback]

passing columns is really only post-processing (a reindex), you are actually reading in the entire store.. You have to read the entire store in; it is a row-oriented store.

It IS possible to only select a single column; use select_column. So if you are relly after only a small number of columns and have a wide-table; you could iterate and select by column then concat.

This is really a data alignment issue; You know you data best and how you are accessing it. It may make sense to store it in multiple tables (see append_as_multiple). possibly store it transposes), or use a higher dim container.

It is possible to do an enhancement to select a smaller number of columns more efficienty (by the method I described above), but I think it would have to be user initiated, as its pretty hard to 'guess' a-prior which method is faster (e.g. suppose you are doing a query which actually only returns a small number of rows, then it is clearly more efficient to do the query once and just select all the columns out and reindex afterwards). - this is essentially what the store is optimized for now.

[CarstVaartjes]

Hi, thanks Jeff; that explains the first part really well (3gb->1gb)!
However, I do not understand the second part where we have a 1gB DataFrame that only uses 100mB if I remake it by extracting the .values and making a new DataFrame from it.

Somewhere a DataFrame generated from HDF5 ends up with 90% overhead that disappears when I extract the numpy arrays, delete the DataFrame and make a new one using the arrays. (see from "a = test_df['a'].values")...

Any idea what might cause that? Because it doesn't seem to be temporary memory usage, waiting to be picked up by the GC (at least not in my own tests)

[jreback]

that's not how python memory works; its really tricky to figure out how much memory something actually takes. Yes the actual memory usage may be calculated using nbytpes but that has nothng to do with the memory you are actually using (well not nothing...)

Its really the peak memory usage that python grabs from the os...but doesn't release it until the process exists.

How do you 'know' that you are actually using more overhead? (FYI
when I read in a store its 2x the actual usage plus some constant) - so I think that you are just seeing the peak (which even though is garbage collected (maybe) make sure that you are releasing references) - but will never be released back to the os.

[CarstVaartjes]

Thanks! The GC does not make me very happy :) (It creates memory issues running it as separate processes in uwsgi and it mostly has to do with the peak here...)

[jreback]

yep.....a way to somewhat minimize this is to use separate / smaller tables, or chunk read
otherwise get more memory!

[jreback]

lmk if you have other questions...but closing for now