Skip to content

Conversation

@lgarrison
Copy link
Collaborator

@lgarrison lgarrison commented Jun 18, 2019

Gridlink has a rather large memory footprint after the min_sep_opt changes. One way to reduce this is by slimming down the cell_pair strut to hold only pair info, and not cell info (#185). But for autocorrelations, there's another factor-of-2 savings possible from the fact that we're presently allocating enough memory to double-count every cell pair, even though we only singly count them. This PR implements that memory saving for autocorrelations.

@manodeep: I think it should be obvious pretty quickly if I messed up the max_num_cell_pairs calculation, since we have an explicit num_cell_pairs < max_num_cell_pairs check. Let's see what Travis says (local tests are passing).

Also feel free to tack on the cell_pair slimming optimizations to this PR.

@lgarrison lgarrison requested a review from manodeep June 18, 2019 14:10
@lgarrison lgarrison added this to the v2.3.1 milestone Jun 18, 2019
@manodeep
Copy link
Owner

@lgarrison Thanks for that version bump - I thought I was missing one version update somewhere.

I am running INTEGRATION_TESTS on the local supercomputer - should take about 48 hours or so to complete. Assuming those pass, we can merge the PR

@manodeep
Copy link
Owner

@lgarrison The INTEGRATION_TESTS have passed. Should we just merge this PR and make a new release?

@lgarrison
Copy link
Collaborator Author

Yes, let's do it!

@lgarrison lgarrison merged commit e401645 into master Jun 21, 2019
@manodeep
Copy link
Owner

Released on both github and PyPI. Should we delete the gridlink-memory.. branch?

@lgarrison
Copy link
Collaborator Author

Yep! I just left it in case you had something left to add.

@lgarrison lgarrison deleted the gridlink-memory-footprint branch June 21, 2019 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants