Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Panel slicing speed #979

wesm opened this issue Mar 29, 2012 · 1 comment


Copy link

commented Mar 29, 2012

from mailing list:

To follow up:

I used timeit.Timer() to time this two different ways:
a) return a slice of a Panel using panel.ix[foo, bar, qux] (where foo,
bar, and qux are arrays)
b) i) iterate through the items of the panel that are in foo
   ii) do a slice on each DataFrame with data.ix[bar, qux]
   iii) save the slice
   iv) reassemble the slices into a new, smaller Panel
   v) return the new Panel

For the data I was using (10 items x 768432 major x 9 minor), the
results I got were:
   t = timeit.Timer(a, setup=setup)
   print min(t.repeat(repeat=3, number=10))

   t = timeit.Timer(b, setup=setup)
   print min(t.repeat(repeat=3, number=10))

This seems odd to me. Approach b is not only faster than a, but almost
5-fold faster. It appears that Panel.ix[foo, bar, qux] is doing
something even slower than the dumb approach embodied in B - either
that or I'm doing something horribly wrong! :-)

wesm added a commit that referenced this issue Jul 13, 2012

This comment has been minimized.

Copy link
Member Author

commented Jul 13, 2012

I was able to speed this up by almost 5x:

Test case:

p = Panel(np.random.randn(10, 768432, 9))
a0 = range(8)
a1 = np.random.permutation(768432)[:100000]
a2 = range(6)
p.ix[a0, a1, a2]


In [2]: timeit p.ix[a0, a1, a2]
1 loops, best of 3: 285 ms per loop


In [2]: timeit p.ix[a0, a1, a2]
10 loops, best of 3: 67.9 ms per loop

@wesm wesm closed this Jul 13, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
1 participant
You can’t perform that action at this time.