Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make pruning much much less conservative #571

Merged
merged 6 commits into from
Mar 9, 2020
Merged

Make pruning much much less conservative #571

merged 6 commits into from
Mar 9, 2020

Conversation

jrapin
Copy link
Contributor

@jrapin jrapin commented Mar 5, 2020

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue

Optimization may slow down dramatically at large number of iterations because the archive becomes big. this makes it much smaller.
The archive is therefore limited to 1000 points most of the time (more for large num_workers), which seems big enough in much cases

How Has This Been Tested (if it applies)

Checklist

  • The documentation is up-to-date with the changes I made.
  • I have read the CONTRIBUTING document and completed the CLA (see CONTRIBUTING).
  • All tests passed, and additional code has been covered with new tests.

@jrapin jrapin requested a review from teytaud March 5, 2020 16:25
@jrapin jrapin self-assigned this Mar 5, 2020
@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Mar 5, 2020
@jrapin
Copy link
Contributor Author

jrapin commented Mar 5, 2020

@teytaud what do you think? I feel the archive is wasting time :s

@teytaud
Copy link
Contributor

teytaud commented Mar 9, 2020

@teytaud what do you think? I feel the archive is wasting time :s

Ok my educated guess, taking into account the fact that the archive is the number of points, not the number of points with their repetitions, is something like "7 times num_workers should be enough" (or 7 times the population size); and possibly much more than that when we apply something like "clearing"...

but I might be completely wrong :-)

@jrapin
Copy link
Contributor Author

jrapin commented Mar 9, 2020

@teytaud what do you think? I feel the archive is wasting time :s

Ok my educated guess, taking into account the fact that the archive is the number of points, not the number of points with their repetitions, is something like "7 times num_workers should be enough" (or 7 times the population size); and possibly much more than that when we apply something like "clearing"...

but I might be completely wrong :-)

If that fits 99% if use case then I go for it. This is only a default value, which can be modified by the optimizer if need be.

@jrapin jrapin added Difficulty: Low Priority: Medium Type: Bug Something isn't working Type: Enhancement New feature or request labels Mar 9, 2020
@jrapin jrapin merged commit 62786f5 into master Mar 9, 2020
@jrapin jrapin deleted the pruning branch March 9, 2020 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed Do not delete this pull request or issue due to inactivity. Difficulty: Low Priority: Medium Type: Bug Something isn't working Type: Enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants