Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Panel slicing speed #979

Closed
wesm opened this issue Mar 29, 2012 · 1 comment

Comments

@wesm
Copy link
Member

commented Mar 29, 2012

from mailing list:


To follow up:

I used timeit.Timer() to time this two different ways:
a) return a slice of a Panel using panel.ix[foo, bar, qux] (where foo,
bar, and qux are arrays)
b) i) iterate through the items of the panel that are in foo
   ii) do a slice on each DataFrame with data.ix[bar, qux]
   iii) save the slice
   iv) reassemble the slices into a new, smaller Panel
   v) return the new Panel

For the data I was using (10 items x 768432 major x 9 minor), the
results I got were:
   t = timeit.Timer(a, setup=setup)
   print min(t.repeat(repeat=3, number=10))
   2.52914

   t = timeit.Timer(b, setup=setup)
   print min(t.repeat(repeat=3, number=10))
   0.54757

This seems odd to me. Approach b is not only faster than a, but almost
5-fold faster. It appears that Panel.ix[foo, bar, qux] is doing
something even slower than the dumb approach embodied in B - either
that or I'm doing something horribly wrong! :-)


Iain
wesm added a commit that referenced this issue Jul 13, 2012
@wesm

This comment has been minimized.

Copy link
Member Author

commented Jul 13, 2012

I was able to speed this up by almost 5x:

Test case:

p = Panel(np.random.randn(10, 768432, 9))
a0 = range(8)
a1 = np.random.permutation(768432)[:100000]
a2 = range(6)
p.ix[a0, a1, a2]

Before:

In [2]: timeit p.ix[a0, a1, a2]
1 loops, best of 3: 285 ms per loop

After:

In [2]: timeit p.ix[a0, a1, a2]
10 loops, best of 3: 67.9 ms per loop

@wesm wesm closed this Jul 13, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.