Ask the evaluator during an epoch, not only at its boundary - #98
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The cheap measures can be driven a long way without the drawing getting better — on one run they improved 64% while the evaluator scored the result within 0.000004 of a run it should have beaten. Asking the evaluator only at the epoch boundary means noticing that after the fact.
What runs
--epoch-eval-interval(default 2000 tasks) puts the front to the evaluator mid-epoch. The field isepoch_parents(pool, FRONT_EVAL_CAP)— the best-ranked distinct candidates, capped at 24, since the top tier can be most of the pool and evaluating near-clones spends the expensive part of the run learning nothing.--epoch-eval-patience(unset) ends the epoch once the evaluator has gone that many rounds without seeing anything better. Rounds rather than checks, so the number means the same thing whatever cadence checks run at.Both are affordable only because of #97: the evaluator's score is absolute and cached per node, so a check re-prices what is new and recalls the rest. A fully cached check does not load a model.
Why patience is unset
Too low and epochs end on the evaluator's noise; too high and it does not stop the unsupervised drift it exists for. Nothing measured so far pins it, and defaulting it on would be guessing — the mistake that put collapse floors in and ended every epoch of a run on arrival.
best_nodenow updates at every check rather than only at boundaries, so the run's best is verified far more often, and the final fallback is a candidate the evaluator has actually seen.Also here
epoch_parentsstill sorted its Pareto front byn.score, which #96 made a constant — a dead sort. Removed:non_dominated_sortalready yields the best tier first, so the head is the best-ranked distinct candidates and there is nothing left to sort by.Not here
The GPU variant. The plan was back-to-back checks on GPU and interval-based on CPU; this is interval-based on both. Running the evaluator on a background thread alongside search needs its own change for shutdown and for the races around a pool that mutates while it is being read — it should not ride along with the criterion itself.