Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.
Sign upFix large concatenations and stacks. #51
Conversation
hameerabbasi
force-pushed the
hameerabbasi:fix-large-concat-stack
branch
from
cf0c89f
to
1f04bd6
Dec 28, 2017
hameerabbasi
force-pushed the
hameerabbasi:fix-large-concat-stack
branch
from
1f04bd6
to
51a1360
Dec 28, 2017
mrocklin
reviewed
Dec 28, 2017
This looks good to me. I had a small question, but please feel free to merge at will.
| data = np.concatenate([x.data for x in arrays]) | ||
| coords = np.concatenate([x.coords for x in arrays], axis=1) | ||
| coords = np.concatenate([x.coords for x in arrays], axis=1).astype(coords_dtype) |
This comment has been minimized.
This comment has been minimized.
mrocklin
Dec 28, 2017
Collaborator
Is it better to call .astype(...) before concatenation on all of arrays or after as is done here?
This comment has been minimized.
This comment has been minimized.
hameerabbasi
Dec 28, 2017
Collaborator
I would think after. Because it would be applied to a bigger array and thus the overhead would be small. Memory would be better, too, as we are just passing already-allocated arrays to concatenate, otherwise we'd have to allocate for each temp array.
Incidentally, astype just returns the original array if it is already the correct dtype.
hameerabbasi
merged commit 9988aec
into
pydata:master
Dec 28, 2017
1 check passed
continuous-integration/travis-ci/pr
The Travis CI build passed
Details
hameerabbasi
deleted the
hameerabbasi:fix-large-concat-stack
branch
Dec 28, 2017
hameerabbasi
referenced this pull request
Dec 31, 2017
Closed
Problems converting dask chunked sparse arrays greater size than 256x256 #32
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
hameerabbasi commentedDec 28, 2017
Addresses #32.