/nbodykit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A few improvements to slicing of a catalog. #543

Merged
merged 4 commits into from Dec 23, 2018

Conversation

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@rainwoodman
@rainwoodman
Copy link
Member

rainwoodman commented Dec 22, 2018

  1. Fixed a bug regarding calculation of total sliced size.
  2. Give ConstantArray a dask name, such that dask does not hang trying to tokenize huge ConstantArray objects.
  3. Use numpy.copy to decouple sliced columns from original data. This improves the memory footprint substantially when slicing is subsampling; previously the new computed array will have references to chunks of the full original data.

@rainwoodman rainwoodman force-pushed the rainwoodman:slicing-copy branch from 628b07e to 0d677fe Dec 22, 2018

@rainwoodman rainwoodman merged commit 7250cfe into bccp:master Dec 23, 2018
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.001%) to 94.662%
Details

@rainwoodman rainwoodman deleted the rainwoodman:slicing-copy branch Dec 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment