Optionally use sys.getsizeof in DataFrame.memory_usage

[mrocklin]

I would like to know how many bytes my dataframe takes up in memory. The standard way to do this is the memory_usage method

df.memory_usage(index=True)

For object dtype columns this measures 8 bytes per element, the size of the reference not the size of the full object. In some cases this significantly underestimates the size of the dataframe.

It might be nice to optionally map sys.getsizeof on object dtype columns to get a better estimate of the size. If this ends up being expensive then it might be good to have this as an optional keyword argument.

df.memory_usage(index=True, measure_object=True)

[jreback]

In [13]: import sys

In [14]: df = DataFrame({'A' : ['foo','bar']*1000000})

In [15]: df['B'] = df['A'].astype('category')

In [16]: df.memory_usage()
Out[16]: 
A    16000000
B     2000016
dtype: int64

In [17]: %timeit df.memory_usage()
10000 loops, best of 3: 66.1 µs per loop

In [18]: %timeit sum([ sys.getsizeof(v) for v in df['A'].values ])
1 loops, best of 3: 467 ms per loop

[mrocklin]

Yup, probably shouldn't be default. I'd be quite happy to opt-in.

On the other side of the comparision, 500ms is very small compared to serialization and communication time if we mistakenly decide that it's a good idea to communicate this dataframe to another machine.

[jreback]

going to give you a cythonized comparison in a sec

[jreback]

In [8]: import sys

In [3]: pd.lib.memory_usage_of_array(np.array(['a'],dtype=object))
Out[3]: 46

In [4]: pd.lib.memory_usage_of_array(np.array(['ab'],dtype=object))
Out[4]: 47

In [5]: pd.lib.memory_usage_of_array(np.array(['abc'],dtype=object))
Out[5]: 48

In [6]: pd.lib.memory_usage_of_array(np.array(['abcd'],dtype=object))
Out[6]: 49

In [7]: df = DataFrame({'A' : ['foo','bar']*1000000})

In [9]: sum([ sys.getsizeof(v) for v in df['A'].values ]) + df['A'].values.nbytes
Out[9]: 96000000

In [10]: pd.lib.memory_usage_of_array(df['A'].values)
Out[10]: 96000000

In [11]: %timeit pd.lib.memory_usage_of_array(df['A'].values)
10 loops, best of 3: 108 ms per loop

In [12]: %timeit sum([ sys.getsizeof(v) for v in df['A'].values ]) + df['A'].values.nbytes
1 loops, best of 3: 481 ms per loop

[jreback]

I am giving back the original nbytes + the actual overhead (e.g. to store it costs you the pointer in the ndarray + the actual storage)

[mrocklin]

Seems like a good idea. The speedup there is nice.

[jreback]

yep, ok, easy enough (then would turn off the '+') if you opt-in

[jreback]

In [7]: df = DataFrame({'A' : ['foo','bar']*1000000})

In [8]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2000000 entries, 0 to 1999999
Data columns (total 1 columns):
A    object
dtypes: object(1)
memory usage: 30.5+ MB

In [9]: df.info(memory_usage='deep')
<class 'pandas.core.frame.DataFrame'>
Int64Index: 2000000 entries, 0 to 1999999
Data columns (total 1 columns):
A    object
dtypes: object(1)
memory usage: 106.8 MB

[MaximilianR]

What do you think about overriding __sizeof__ with one of these?

[jreback]

ahh, so sys.getsizeof(DataFrame) gives a nice answer, sure. that would be ok.

[jickersville]

Glad to finally see #8578 implemented. 👍 . It appears that when a Continuum co-worker complains of a pandas wart it gets fixed in 60 minutes instead of being repeatedly deflected with excuses over the course of 3 days until the user runs away screaming in exasperation.

Good work!

@shoyer @jorisvandenbossche

[jreback]

@jickersville that's not a very nice comment.

What issue has:

being repeatedly deflected with excuses over the course of 3 days until the user runs away screaming in exasperation.

????

[jreback]

since @jickersville account was created today. I suspect you are actually @kay1793 whom was banned for egregious behavior. prove me wrong here.

[shoyer]

sigh

On Wed, Mar 30, 2016 at 10:23 AM, Jeff Reback notifications@github.com
wrote:

since @jickersville https://github.com/jickersville account was created
today. I suspect you are actually @kay1793 https://github.com/kay1793

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
pydata#11595 (comment)