-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ML] Line search for best learn rate #948
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good work altogether! I am wondering if adding mainLoop
as prefix or suffix actually improves readability since very often there is no alternative variable, like mainLoopMaximumNumberTrees
and initializationLoopMaximumNumberTrees
. The exception beeing, totalNumberSteps
and mainLoopNumberSteps
.
Thanks for the review @valeriy42. I've now addressed all your comments can you take another look? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Good work on providing explanations!
This switches to performing an initial line search over the learn rate and reducing the range we search in the main optimisation loop. This does cost us some run time, but we generally get small wins in both the QoR and the stability of QoR as borne out by the sample results below.
Because we can't estimate the runtime upfront I've extended the progress monitor to allow us to reset the expected total number of steps. This will advance progress if it needs to; we have no mechanism for reducing progress if the expected number steps increase. In this case, we simply get stuck at the current progress. I use a margin in the initial estimates to reduce the chance of this so we get smoother progress monitoring.
Sample regression results:
Sample classification results: