Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

optimize count for groupby #132

Open
dcherian opened this issue Oct 2, 2023 · 3 comments · May be fixed by #159
Open

optimize count for groupby #132

dcherian opened this issue Oct 2, 2023 · 3 comments · May be fixed by #159

Comments

@dcherian
Copy link
Contributor

dcherian commented Oct 2, 2023

Surfacing Stephan's comment from #13 (comment)

Note that we could probably improve the speed of existing methods by allocating the counts array once, outside the gufunc's inner loop.

which I think is referring to

numbagg/numbagg/grouped.py

Line 20 in 592da9c

counts = np.zeros(out.shape, dtype=labels.dtype)

@dcherian
Copy link
Contributor Author

dcherian commented Oct 2, 2023

Another interesting optimization would be that the max needed value for count is the prod of sizes along axis, so technically you could get away with a much smaller int

@max-sixty
Copy link
Collaborator

Yes, good idea.

IIUC this would help where there's a multi-dimensional output. Otherwise it's a single allocation anyway...

@max-sixty
Copy link
Collaborator

For transparency — I'm pushing these off for the moment. We're already 2x faster than pandas, and so we can wait until we have more adoption before adding optimizations.

(Though if someone in the future reads this — contributions welcome!)

dcherian added a commit to dcherian/numbagg that referenced this issue Oct 15, 2023
@dcherian
Optimize memory allocation for counts
a235be8 
Closes numbagg#132
dcherian added a commit to dcherian/numbagg that referenced this issue Oct 15, 2023
@dcherian
Optimize memory allocation for counts
2a3c8eb 
Closes numbagg#132
dcherian added a commit to dcherian/numbagg that referenced this issue Oct 15, 2023
@dcherian
Optimize memory allocation for counts
dbaec95 
Closes numbagg#132
@dcherian dcherian linked a pull request Oct 15, 2023 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Optimize memory allocation for counts dcherian/numbagg
2 participants
@dcherian @max-sixty