frame values access is slow with large frames with different dtypes

[cpcloud]

In addtion, there is a memory leak. Repeatedly evaling big.values eats up the amount of RAM taken up by the array.

from pandas import DataFrame, concat
from numpy.random import randint, randn
from string import ascii_letters as letters
ncols = 16
nrows = 1e7
letters = list(letters)
df = DataFrame(randn(nrows, ncols), columns=letters[:ncols])
col = DataFrame(randint(1, size=nrows), columns=[letters[ncols]])
big = concat([df, col], axis=1)
big.values  # very slow and leaky

[jreback]

this is by definition slow
a lot of copying to form a common dtype
it's not a good idea to do this

can I reproduce the leak?

[cpcloud]

haha, i think only u can answer that.

[jreback]

i meant u!

I'll take a look

[cpcloud]

yep the leak occurs just starting a fresh ipython session

[cpcloud]

ipython is the culprit for at least the display delay, for sure. there's some code that reruns the last expression so in fact values does get called twice. will raise the issue over there.

[cpcloud]

@jreback If u put ipdb; ipdb.set_trace() on line 1424 of internals.py and run my example and then press c in the debugger you'll see that u must hit c twice.

[jreback]

ah u have a duplicate frame columns across dtypes
I should have put a warning
this is not fast as it is not that common
let me see if I can fix up

[cpcloud]

my mistake. still happens without dups (didn't mean to have dups to begin with). i changed the example above

[jreback]

Accessing the number array, no copy

In [54]: %timeit df.values
100000 loops, best of 3: 3.47 us per loop

Copy with dtype consolidate across dtypes

In [55]: %timeit big.values
1 loops, best of 3: 3.32 s per loop

Copy but not consolidation

In [56]: %timeit big.copy()
1 loops, best of 3: 546 ms per loop

The inner loop (w/o reindexing)

In [57]: %timeit big._data._interleave(big._data.items)
1 loops, best of 3: 3.31 s per loop

The reindexing step

In [58]: %timeit big._data.reindex_items(big._data.items)
1 loops, best of 3: 499 ms per loop

This is a time consuming step, copying float data to object dtype
that is the fundamental problem
Manually doing the _interleave

In [59]: x = np.empty(df.shape,dtype='object')

In [60]: df._data.items.get_indexer(df._data.items)
Out[60]: array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15])

In [61]: x[np.arange(len(df.values))] = df.values

In [63]: %timeit x[np.arange(len(df.values))] = df.values
1 loops, best of 3: 8.26 s per loop

Here is a quick way (no copying)

In [64]: %time big.blocks
CPU times: user 0.29 s, sys: 0.22 s, total: 0.51 s
Wall time: 0.52 s
Out[64]: 
{'float64': <class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Columns: 16 entries, a to p
dtypes: float64(16),
 'int64': <class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Columns: 1 entries, q to q
dtypes: int64(1)}

In [65]: %timeit big.blocks
1 loops, best of 3: 547 ms per loop

[cpcloud]

@jreback nice. should i raise over at ipython? still seems like a problem to eval the code twice...also not sure i understand the object dtype part. big has 16 float64 Series and 1 int64 Series. in the second-to-last block of code u use an object array why not just broadcast block type? obvs object dtype arrays will be drastically slower than the numpy scalar types

[jreback]

where is it evaling twice?

[jreback]

The copy from a float/int dtype type to an object array (that is bigger) is the cost here;
prob because it gets copied in a non-optimal way, maybe better to flatten then reshape in the copy (instead of a 2-dim copy); again this is only an issue of merging dtypes (where one is object); normally you don't do this very often (or shouldn't), for this reason

[cpcloud]

If you put a break point right before interleave is called you must type
continue twice to return to normal ipython from the debugger.
On May 17, 2013 8:35 AM, "jreback" notifications@github.com wrote:

where is it evaling twice?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3629#issuecomment-18058818
.

[cpcloud]

Ipython evals the ast of a cell
On May 17, 2013 8:39 AM, "Phillip Cloud" cpcloud@gmail.com wrote:

If you put a break point right before interleave is called you must type
continue twice to return to normal ipython from the debugger.
On May 17, 2013 8:35 AM, "jreback" notifications@github.com wrote:

where is it evaling twice?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3629#issuecomment-18058818
.

[cpcloud]

I suspect it lies somewhere in run_cell in core ipython can't remember the filename I'm typing this on my phone sorry