Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unstack is not careful about potential memory use problems #2278

Closed
wesm opened this issue Nov 17, 2012 · 0 comments

Comments

@wesm
Copy link
Member

commented Nov 17, 2012

Cartesian product problem; unstacks relative to hypothetical possibilities instead of observed combinations. I'm claiming this unless someone else wants to look inside the reshape code

import pandas as pd
import numpy as np

# Generate Long File & Test Pivot
NUM_ROWS = 1000000

df = pd.DataFrame({'A' : np.random.randint(100, size=NUM_ROWS), 
                                'B' : np.random.randint(300, size=NUM_ROWS), 
                                'C' : np.random.randint(-7, 7, size=NUM_ROWS), 
                                'D' : np.random.randint(-19,19, size=NUM_ROWS),
                                'E' : np.random.randint(3000, size=NUM_ROWS),
                                'F' : np.random.randn(NUM_ROWS)})

df_pivoted = df.pivot_table(rows=['A', 'B', 'C'], cols='E', values='F')
df_pivoted
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.