-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/podaac 5065 #49
Conversation
It's an interesting solution, but it seems kinda inefficient. Is there a way to determine whether or not merging should run in single-core vs multi-core before the run? Multi-core merging should be fairly deterministic in terms of how much memory is consumed. It's essentially the sum of each variable's |
Alternatively, the merge workers are limited by the number of variables that can fit in the queue. Could calculate the maximum number of variables that can fit in the queue for a given amount of free space and that would allow for similar multi-core performance while keeping the memory usage below a certain level |
thanks for suggestions josh, i made modification where i calculate the size that will be needed by variables to determine if it will fit within the shared memory, and from there determine if we can do multicore or single core. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you just update the PR description to match the end result of the change?
Description
Integration with SMAP_RSS_L2_SSS_V5 collection with concise
Overview of work done
Calculated how much memory a variable will need in shared memory, determine if the variable will fit in shared memory and run single core or multicore concise accordingly.
Overview of verification done
Tested locally with SMAP_RSS_L2_SSS_V5 collection.
Overview of integration done
Explain how this change was integration tested. Provide screenshots or logs if appropriate. An example of this would be a local Harmony deployment.
PR checklist:
See Pull Request Review Checklist for pointers on reviewing this pull request