[ML] Stochastic derivatives for boosted tree training #811
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
valeriy42
reviewed
Nov 14, 2019
|
|
||
| // We need to scale the regularisation terms to account for the difference | ||
| // in the down sample factor compared to the value used in the line search. | ||
| auto scaleRegularizers = [&](CBoostedTreeImpl& tree, double downsampleFactor) { |
There was a problem hiding this comment.
Good thinking of need to scale the regularizers! ➕
Comment on lines
+692
to
+693
| nextTreeCountToRefreshSplits += | ||
| static_cast<std::size_t>(std::max(0.5 / eta, 2.0)); |
There was a problem hiding this comment.
It would be nice to have a comment explaining what is going on here.
There was a problem hiding this comment.
I added a catch all comment for amortising cost computing splits. See 56f32ea. I didn't explicitly mention why I tied to eta, i.e. captures something about how different we expect splits chosen from round-to-round to be. Does this seem sufficient to you?
|
Thanks for the review @valeriy42 and good catch on missing persistence changes! I think I've now addressed all your comments. Can you take another look? |
valeriy42
approved these changes
Nov 14, 2019
tveasey
changed the title
tveasey
added a commit
to tveasey/ml-cpp-1
that referenced
this pull request
Nov 28, 2019
tveasey
added a commit
that referenced
this pull request
Nov 28, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The primary change is to use a random downsample (bag) of data for each new tree we train. I extended initialisation to perform a line search over the downsampling ratio to find a good range to search during fine tuning. This is primarily intended to improve runtime on larger data sets, but incidentally improved QoR for many data sets since it tends to decorrelates the errors between trees.
This also tweaks the initial range used for line search to base its start and end point on percentile values for the gain and total curvature of a tree. Finally, since I needed to update progress monitoring to account for the extra line searches, I switched to updating progress after each tree which is trained. This gives much finer grained progress monitoring.