Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize sparse COO matrix sum #747

Closed
liuzhenqi77 opened this issue Sep 22, 2022 · 2 comments
Closed

Optimize sparse COO matrix sum #747

liuzhenqi77 opened this issue Sep 22, 2022 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@liuzhenqi77
Copy link
Contributor

Summary

When trying to speed up MKDAChi2(), I noticed COO matrix sum takes a lot of time.

Additional details

Examples here and here. It seems to me that the implementation of sum in sparse only uses np.add.reduce without any optimization (I opened an issue there waiting for response).

Next steps

I tried some simple optimization like below using numba, which makes it much faster. (There are many better ways to implement this. Also maybe there are ways to avoid calculating the sum.)

from numba import njit

@njit
def _coo_sum_axis_0(shape, coords, data):
    res = np.zeros(shape)
    for i in range(len(data)):
        x, y, z = coords[1:, i]
        res[x, y, z] += data[i]
    return res

# _coo_sum_axis_0(ma_maps.shape[1:], ma_maps.coords, ma_maps.data)
@liuzhenqi77 liuzhenqi77 added the enhancement New feature or request label Sep 22, 2022
@JulioAPeraza
Copy link
Collaborator

Thank you for noticing this issue, @liuzhenqi77. We think that other functions may be suffering from the same issue as well, for example

# Need to convert to dense because np.ceil is too slow with sparse
max_ma_values = ma_maps.max(axis=[1, 2, 3]).todense()

where max also uses np.maximum.reduce.

We will submit a PR with a solution to this soon.
Thanks!

@JulioAPeraza JulioAPeraza self-assigned this Dec 8, 2022
@JulioAPeraza
Copy link
Collaborator

I think we can close this since #857 addressed this performance issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants