Loading…
MRG: GBRT loss-functions #924
4 participants
I tried running the example and it seems to never complete: I launched the process more that 1min ago and it should have converged on such a small dataset (infinite loop?). I cannot stop it with a keyboard interruption to see the python traceback as it's in the compiled cython code.
very strange - did you run make? could you send me your configuration (numpy, scipy)? thx
Yes I ran make on pprett/gbrt-loss. I am running on python 2.6 from an old 2009 mac osx laptop with numpy 1.6.1 and scipy 0.10.1.
I tried to run it under gdb and the script run fine in less than 10s ...
hmm... the example fits 3 gbrt models (250 iterations and 100 samples) and predicts the mean, upper and lower quantile on 1000 samples - so it should run in a couple of seconds.
maybe there is another dtype issue in the cython code that I haven't covered yet - I'll add some more tests to nail it down.
could you do me a favor and re-run the example using the latest commit (cf09064) and check if the error occurs?
thanks, p
Alright false alarm.... It was just that the matplotlib window was opening in the background (behind my other open windows)...
Sorry for the noise...
Alright, I had a quick look at the code and it seems fine although I don't know much about GBRT. +1 for merging from my side.
np - thanks for checking!
| @@ -0,0 +1,78 @@ | ||
| +""" | ||
| +===================================================== | ||
| +Prediction Intervals for Gradient Boosting Regression | ||
| +===================================================== | ||
| + | ||
| +This example shows how quantile regression can be used | ||
| +to create prediction intervals. | ||
| +""" | ||
| + | ||
| +import numpy as np | ||
| +from sklearn.ensemble import GradientBoostingRegressor | ||
| +from matplotlib import pyplot as pl |
|
got converted to pyplot? :) changed that - thx
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
|
| @@ -0,0 +1,78 @@ | ||
| +""" | ||
| +===================================================== | ||
| +Prediction Intervals for Gradient Boosting Regression | ||
| +===================================================== | ||
| + | ||
| +This example shows how quantile regression can be used | ||
| +to create prediction intervals. | ||
| +""" | ||
| + | ||
| +import numpy as np | ||
| +from sklearn.ensemble import GradientBoostingRegressor | ||
| +from matplotlib import pyplot as pl | ||
| + | ||
| +np.random.seed(1) |
|
do we want global seeding in the examples? @agramfort I've copied the data generation from
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
|
Code looks good. All tests pass on my machine. +1 for merge.
The example is nice! However I was wondering, how does it compare with Gaussian Processes? We actually have the same example for GP, maybe merging them should be better?
+1 for combining GP and quantile GBRT examples.
regarding the example: do you think its better to merge them into one example or should the GP example remain unchanged and I include GP in the quantile example?
+1 for mergning them and having one less example to maintain.
after looking at the GP example in more detail I don't think its a good idea to merge it with the quantile example - Vincent and Jake have done a great job in explaining the effect of some of the hyper-parameters on the GP posterior - IMHO adding the quantile regression plot would not contribute to the purpose of the example.
I agree that too much examples are a maintenance burden thus I decided to remove the quantile regression example. I don't think that it is of interest to a broader audience anyways - I rather think that I create a gist from the code and make a short blog post.
Please keep the quantile regression as it is, as a separate example then and maybe cross-link the two examples as they are completely different solution to the same kind of problems.
BTW: looking forward to reading your blog post too :)
Good. I am +1 for merge.
+1 to keep the quantile regression example.
you should also update the what's new
@ogrisel @glouppe @agramfort thanks for the reviews!
-
-
Conflicts: sklearn/ensemble/gradient_boosting.py
-
…vals; adopted the GP regression example to show-case prediction intervals
-
-
Conflicts: sklearn/tree/_tree.c sklearn/tree/_tree.pyx
-
Conflicts: sklearn/tree/_tree.c
- 22 10 doc/modules/ensemble.rst
- 10 3 doc/whats_new.rst
- 79 0 examples/ensemble/plot_gradient_boosting_quantile.py
- 15 9 examples/ensemble/plot_gradient_boosting_regularization.py
- 0 1 sklearn/ensemble/__init__.py
- 165 18 sklearn/ensemble/gradient_boosting.py
- 97 1 sklearn/ensemble/tests/test_gradient_boosting.py
- 1 1 sklearn/tree/_tree.c
| @@ -0,0 +1,79 @@ | ||
| +""" | ||
| +===================================================== | ||
| +Prediction Intervals for Gradient Boosting Regression | ||
| +===================================================== | ||
| + | ||
| +This example shows how quantile regression can be used | ||
| +to create prediction intervals. | ||
| +""" | ||
| + | ||
| +import numpy as np | ||
| +import pylab as pl | ||
| +from sklearn.ensemble import GradientBoostingRegressor | ||
| + | ||
| + | ||
| +np.random.seed(1) | ||
| + | ||
| + | ||
| +def f(x): | ||
| + """The function to predict.""" | ||
| + return x * np.sin(x) | ||
| + | ||
| +#---------------------------------------------------------------------- | ||
| +# First the noiseless case | ||
| +X = np.atleast_2d(np.random.uniform(0, 10.0, size=100)).T | ||
| +X = X.astype(np.float32) | ||
| + | ||
| +# Observations | ||
| +y = f(X).ravel() | ||
| + | ||
| +dy = 1.5 + 1.0 * np.random.random(y.shape) | ||
| +noise = np.random.normal(0, dy) | ||
| +y += noise | ||
| +y = y.astype(np.float32) | ||
| + | ||
| +# Mesh the input space for evaluations of the real function, the prediction and | ||
| +# its MSE | ||
| +xx = np.atleast_2d(np.linspace(0, 10, 1000)).T | ||
| +xx = xx.astype(np.float32) | ||
| + | ||
| +alpha = 0.95 | ||
| + | ||
| +clf = GradientBoostingRegressor(loss='quantile', alpha=alpha, | ||
| + n_estimators=250, max_depth=3, | ||
| + learn_rate=.1, min_samples_leaf=9, | ||
| + min_samples_split=9) | ||
| + | ||
| +clf.fit(X, y) | ||
| + | ||
| +# Make the prediction on the meshed x-axis | ||
| +y_upper = clf.predict(xx) | ||
| + | ||
| +clf.set_params(alpha=1.0 - alpha) | ||
| +clf.fit(X, y) | ||
| + | ||
| +# Make the prediction on the meshed x-axis | ||
| +y_lower = clf.predict(xx) | ||
| + | ||
| +clf.set_params(loss='ls') | ||
| +clf.fit(X, y) | ||
| + | ||
| +# Make the prediction on the meshed x-axis | ||
| +y_pred = clf.predict(xx) | ||
| + | ||
| +# Plot the function, the prediction and the 95% confidence interval based on | ||
| +# the MSE | ||
| +fig = pl.figure() | ||
| +pl.plot(xx, f(xx), 'g:', label=u'$f(x) = x\,\sin(x)$') | ||
| +pl.plot(X, y, 'b.', markersize=10, label=u'Observations') | ||
| +pl.plot(xx, y_pred, 'r-', label=u'Prediction') | ||
| +pl.plot(xx, y_upper, 'k-') | ||
| +pl.plot(xx, y_lower, 'k-') | ||
| +pl.fill(np.concatenate([xx, xx[::-1]]), | ||
| + np.concatenate([y_upper, y_lower[::-1]]), | ||
| + alpha=.5, fc='b', ec='None', label='95% prediction interval') | ||
| +pl.xlabel('$x$') | ||
| +pl.ylabel('$f(x)$') | ||
| +pl.ylim(-10, 20) | ||
| +pl.legend(loc='upper left') | ||
| +pl.show() |
Two new regression loss functions for gradient boosting:
Some other gimmicks:
max_featuresfor variance reduction (similar toRandomForest).