Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve ensemble selection memory usage #997

Merged
merged 4 commits into from Nov 5, 2020

Conversation

mfeurer
Copy link
Contributor

@mfeurer mfeurer commented Nov 5, 2020

  • separate storage of data inside the ensemble builder into two dictionaries to separately store them on disk (one for scores and one for predictions). If we run over the allocated memory, we can still make use of the stored scores
  • do not stop the ensemble builder when the number of models to consider can no longer be reduced so that the ensemble builder can still delete models from the hard drive if necessary.
  • avoid unnecessary memory copies during ensemble construction
  • improve structure of temporary directories during unit testing

* separate storage of data inside the ensemble builder into two dictionaries to separately store them on disk (one for scores and one for predictions). If we run over the allocated memory, we can still make use of the stored scores
* do not stop the ensemble builder when the number of models to consider can no longer be reduced so that the ensemble builder can still delete models from the hard drive if necessary.
* avoid unnecessary memory copies during ensemble construction
* improve structure of temporary directories during unit testing
@mfeurer mfeurer merged commit d3aafbd into development Nov 5, 2020
@mfeurer mfeurer deleted the ensemble_selection_memory branch November 5, 2020 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant