Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+2] Adding return_std options for models in linear_model/bayes.py #7838

Merged
merged 32 commits into from Dec 1, 2016

Conversation

@sergeyf
Copy link
Contributor

@sergeyf sergeyf commented Nov 7, 2016

Reference Issue

The reason for this pull request appears in a conversation for #4844

What does this implement/fix? Explain your changes.

This is the first of two pull requests. The ultimate goal is to add the MICE imputation algorithm to scikit-learn. To do so, we need sklearn's Bayesian regression algorithms to be able to return standard deviations as well as predictions.

This pull requests adds the option return_std to the predict methods of both BayesianRidge and ARDRegression.

Any other comments?

Once this is accepted, I will make a pull request that implements MICE using BayesianRidge by default (which seems more robust to small sample sizes than ARD in my limited experience).

X : {array-like, sparse matrix}, shape = (n_samples, n_features)
Samples.
predict_std : boolean, optional

This comment has been minimized.

@jnothman

jnothman Nov 7, 2016
Member

Please use return_std like GaussianProcessRegressor, and check its docstring for consistency.

This comment has been minimized.

@sergeyf

sergeyf Nov 7, 2016
Author Contributor

Oh my bad - I'll change that to return_std.

@sergeyf sergeyf closed this Nov 7, 2016
@sergeyf sergeyf reopened this Nov 7, 2016
@sergeyf sergeyf changed the title [MRG] Adding predict_std options for models in linear_model/bayes.py [MRG] Adding return_std options for models in linear_model/bayes.py Nov 7, 2016
@sergeyf
Copy link
Contributor Author

@sergeyf sergeyf commented Nov 7, 2016

@jnothman I changed all instances of predict_std to return_std. Thanks for pointing that one out.

@jnothman
Copy link
Member

@jnothman jnothman commented Nov 7, 2016

Could you put the new figure in the fourth quadrant of the existing plots?

@sergeyf
Copy link
Contributor Author

@sergeyf sergeyf commented Nov 7, 2016

The problem is that it the uncertainty grows linearly. I tried zooming out further, but the plot looks qualitatively the same.

One idea is to have a non-linear function f(x), and then use a polynomial kernel to estimate it. I'll give this a shot.

And yes, I'll put the figures in the fourth quadrant.

Thanks.

@sergeyf
Copy link
Contributor Author

@sergeyf sergeyf commented Nov 8, 2016

@jnothman I tried the polynomial regression thing. Check it out. It works better with BayesianRidge than with ARD, but the idea is there in both I think.

@jnothman
Copy link
Member

@jnothman jnothman commented Nov 8, 2016

ARD:
ARD
BRR:
BRR

Inserted with:

ARD:
![ARD](https://$CIRCLE_BUILDNO-843222-gh.circle-artifacts.com/0/home/ubuntu/scikit-learn/doc/_build/html/stable/_images/sphx_glr_plot_ard_004.png)
BRR:   
![BRR](https://$CIRCLE_BUILDNO-843222-gh.circle-artifacts.com/0/home/ubuntu/scikit-learn/doc/_build/html/stable/_images/sphx_glr_plot_bayesian_ridge_004.png)
y_mean, y_std = clf_poly.predict(np.vander(X_plot, degree), return_std=True)
plt.figure(figsize=(6, 5))
plt.errorbar(X_plot, y_mean, y_std, color='navy',
label="Polynomial Bayesian Ridge Regression", linewidth=2)

This comment has been minimized.

@jnothman

jnothman Nov 8, 2016
Member

Incorrect label for ARD

Sergey Feldman added 2 commits Nov 8, 2016
@jnothman
Copy link
Member

@jnothman jnothman commented Nov 20, 2016

I agree those test failures don't appear to be your problem.

@lesteve
Copy link
Member

@lesteve lesteve commented Nov 21, 2016

@sergeyf you still have flake8 errors in Travis. My advice is to take some time to have on-the-fly flake8 linting in your editor of choice, it will save you a lot of time in the long run.

@sergeyf
Copy link
Contributor Author

@sergeyf sergeyf commented Nov 21, 2016

@lesteve Thanks for the suggestion. I use Spyder - I think it has pyflakes integration. I'll dig around. The last two should be fixed now in any case.

Copy link
Member

@amueller amueller left a comment

Looks ok, haven't checked the math (yet?)

@@ -144,6 +157,8 @@ def fit(self, X, y):
X, y = check_X_y(X, y, dtype=np.float64, y_numeric=True)
X, y, X_offset, y_offset, X_scale = self._preprocess_data(
X, y, self.fit_intercept, self.normalize, self.copy_X)
self.X_offset = X_offset

This comment has been minimized.

@amueller

amueller Nov 23, 2016
Member

Should be X_offset_ and X_scale_

@@ -216,10 +231,43 @@ def fit(self, X, y):
self.alpha_ = alpha_
self.lambda_ = lambda_
self.coef_ = coef_
sigma_ = np.dot(Vh.T,
Vh / (eigen_vals_ + lambda_ / alpha_)[:, None])

This comment has been minimized.

@amueller

amueller Nov 23, 2016
Member

I prefer np.newaxis over None


self._set_intercept(X_offset, y_offset, X_scale)
return self

def predict(self, X, return_std=False):
"""Predict using the linear model. In addition to the mean of the

This comment has been minimized.

@amueller

amueller Nov 23, 2016
Member

This is not pep 257: https://www.python.org/dev/peps/pep-0257/

Please add an empty line after the first sentence.

@@ -434,3 +495,34 @@ def fit(self, X, y):
self.lambda_ = lambda_
self._set_intercept(X_offset, y_offset, X_scale)
return self

def predict(self, X, return_std=False):
"""Predict using the linear model. In addition to the mean of the

This comment has been minimized.

@amueller

amueller Nov 23, 2016
Member

Same comment about docstring pep.

return np.dot(X, w) + b

def f_noise(X, noise_mult):
return f(X) + np.random.randn(X.shape[0])*noise_mult

This comment has been minimized.

@amueller

amueller Nov 23, 2016
Member

is that pep8 without space around *? hm

This comment has been minimized.

This comment has been minimized.

n_train = 50
n_test = 10

noise_mult = 0.1

This comment has been minimized.

@amueller

amueller Nov 23, 2016
Member

does this work if you do a for-loop over multiple noise_mult?

This comment has been minimized.

@sergeyf

sergeyf Nov 24, 2016
Author Contributor

Works fine for Bayesian Ridge, but unfortunately, ARD behaves oddly because it gets rid of a bunch of dimensions. It gets it MOSTLY right. If you set noise_mult = 1.0, ARD will get 1.2 or 1.1, which is good enough for the estimation of standard deviation of noise, but I'd have to bump up decimal=0 for it to still pass the tests for such a large noise. I'll do that for the next commit unless you have another suggestion.

@@ -56,3 +56,55 @@ def test_toy_ard_object():
# Check that the model could approximately learn the identity function
test = [[1], [3], [4]]
assert_array_almost_equal(clf.predict(test), [1, 3, 4], 2)


def test_return_std_bayesian():

This comment has been minimized.

@amueller

amueller Nov 23, 2016
Member

If the tests are the same (might have overlooked something?), why not do them both in the same test?

Sergey Feldman
@sergeyf
Copy link
Contributor Author

@sergeyf sergeyf commented Nov 24, 2016

@amueller I've made the requested fixes. Thanks!

@amueller
Copy link
Member

@amueller amueller commented Nov 30, 2016

can you fix pep8? Integration is failing.

@amueller
Copy link
Member

@amueller amueller commented Nov 30, 2016

Other than that LGTM

@amueller amueller changed the title [MRG+1] Adding return_std options for models in linear_model/bayes.py [MRG+2] Adding return_std options for models in linear_model/bayes.py Nov 30, 2016
@sergeyf
Copy link
Contributor Author

@sergeyf sergeyf commented Nov 30, 2016

Thanks @amueller, just fixed that last one.

R. Salakhutdinov, Lecture notes on Statistical Machine Learning,
http://www.utstat.toronto.edu/~rsalakhu/sta4273/notes/Lecture2.pdf#page=15
Their beta is our self.beta_

This comment has been minimized.

@lesteve

lesteve Dec 1, 2016
Member

beta_ is not listed in the attributes, did you mean another attribute maybe?

This comment has been minimized.

@sergeyf

sergeyf Dec 1, 2016
Author Contributor

Thanks for the catch. I meant self.alpha_.

R. Salakhutdinov, Lecture notes on Statistical Machine Learning,
http://www.utstat.toronto.edu/~rsalakhu/sta4273/notes/Lecture2.pdf#page=15
Their beta is our self.beta_

This comment has been minimized.

@lesteve

lesteve Dec 1, 2016
Member

Same remark about beta_ not being listed in the attributes.

@amueller
Copy link
Member

@amueller amueller commented Dec 1, 2016

thanks!

@amueller amueller merged commit 67a85b8 into scikit-learn:master Dec 1, 2016
2 of 3 checks passed
2 of 3 checks passed
continuous-integration/appveyor/pr AppVeyor build failed
Details
ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@sergeyf sergeyf deleted the sergeyf:MICE branch Dec 2, 2016
sergeyf added a commit to sergeyf/scikit-learn that referenced this pull request Feb 28, 2017
…scikit-learn#7838)

* initial commit for return_std

* initial commit for return_std

* adding tests, examples, ARD predict_std

* adding tests, examples, ARD predict_std

* a smidge more documentation

* a smidge more documentation

* Missed a few PEP8 issues

* Changing predict_std to return_std #1

* Changing predict_std to return_std scikit-learn#2

* Changing predict_std to return_std scikit-learn#3

* Changing predict_std to return_std final

* adding better plots via polynomial regression

* trying to fix flake error

* fix to ARD plotting issue

* fixing some flakes

* Two blank lines part 1

* Two blank lines part 2

* More newlines!

* Even more newlines

* adding info to the doc string for the two plot files

* Rephrasing "polynomial" for Bayesian Ridge Regression

* Updating "polynomia" for ARD

* Adding more formal references

* Another asked-for improvement to doc string.

* Fixing flake8 errors

* Cleaning up the tests a smidge.

* A few more flakes

* requested fixes from Andy

* Mini bug fix

* Final pep8 fix

* pep8 fix round 2

* Fix beta_ to alpha_ in the comments
Sundrique added a commit to Sundrique/scikit-learn that referenced this pull request Jun 14, 2017
…scikit-learn#7838)

* initial commit for return_std

* initial commit for return_std

* adding tests, examples, ARD predict_std

* adding tests, examples, ARD predict_std

* a smidge more documentation

* a smidge more documentation

* Missed a few PEP8 issues

* Changing predict_std to return_std scikit-learn#1

* Changing predict_std to return_std scikit-learn#2

* Changing predict_std to return_std scikit-learn#3

* Changing predict_std to return_std final

* adding better plots via polynomial regression

* trying to fix flake error

* fix to ARD plotting issue

* fixing some flakes

* Two blank lines part 1

* Two blank lines part 2

* More newlines!

* Even more newlines

* adding info to the doc string for the two plot files

* Rephrasing "polynomial" for Bayesian Ridge Regression

* Updating "polynomia" for ARD

* Adding more formal references

* Another asked-for improvement to doc string.

* Fixing flake8 errors

* Cleaning up the tests a smidge.

* A few more flakes

* requested fixes from Andy

* Mini bug fix

* Final pep8 fix

* pep8 fix round 2

* Fix beta_ to alpha_ in the comments
NelleV added a commit to NelleV/scikit-learn that referenced this pull request Aug 11, 2017
…scikit-learn#7838)

* initial commit for return_std

* initial commit for return_std

* adding tests, examples, ARD predict_std

* adding tests, examples, ARD predict_std

* a smidge more documentation

* a smidge more documentation

* Missed a few PEP8 issues

* Changing predict_std to return_std #1

* Changing predict_std to return_std #2

* Changing predict_std to return_std #3

* Changing predict_std to return_std final

* adding better plots via polynomial regression

* trying to fix flake error

* fix to ARD plotting issue

* fixing some flakes

* Two blank lines part 1

* Two blank lines part 2

* More newlines!

* Even more newlines

* adding info to the doc string for the two plot files

* Rephrasing "polynomial" for Bayesian Ridge Regression

* Updating "polynomia" for ARD

* Adding more formal references

* Another asked-for improvement to doc string.

* Fixing flake8 errors

* Cleaning up the tests a smidge.

* A few more flakes

* requested fixes from Andy

* Mini bug fix

* Final pep8 fix

* pep8 fix round 2

* Fix beta_ to alpha_ in the comments
paulha added a commit to paulha/scikit-learn that referenced this pull request Aug 19, 2017
…scikit-learn#7838)

* initial commit for return_std

* initial commit for return_std

* adding tests, examples, ARD predict_std

* adding tests, examples, ARD predict_std

* a smidge more documentation

* a smidge more documentation

* Missed a few PEP8 issues

* Changing predict_std to return_std #1

* Changing predict_std to return_std scikit-learn#2

* Changing predict_std to return_std scikit-learn#3

* Changing predict_std to return_std final

* adding better plots via polynomial regression

* trying to fix flake error

* fix to ARD plotting issue

* fixing some flakes

* Two blank lines part 1

* Two blank lines part 2

* More newlines!

* Even more newlines

* adding info to the doc string for the two plot files

* Rephrasing "polynomial" for Bayesian Ridge Regression

* Updating "polynomia" for ARD

* Adding more formal references

* Another asked-for improvement to doc string.

* Fixing flake8 errors

* Cleaning up the tests a smidge.

* A few more flakes

* requested fixes from Andy

* Mini bug fix

* Final pep8 fix

* pep8 fix round 2

* Fix beta_ to alpha_ in the comments
maskani-moh added a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017
…scikit-learn#7838)

* initial commit for return_std

* initial commit for return_std

* adding tests, examples, ARD predict_std

* adding tests, examples, ARD predict_std

* a smidge more documentation

* a smidge more documentation

* Missed a few PEP8 issues

* Changing predict_std to return_std scikit-learn#1

* Changing predict_std to return_std scikit-learn#2

* Changing predict_std to return_std scikit-learn#3

* Changing predict_std to return_std final

* adding better plots via polynomial regression

* trying to fix flake error

* fix to ARD plotting issue

* fixing some flakes

* Two blank lines part 1

* Two blank lines part 2

* More newlines!

* Even more newlines

* adding info to the doc string for the two plot files

* Rephrasing "polynomial" for Bayesian Ridge Regression

* Updating "polynomia" for ARD

* Adding more formal references

* Another asked-for improvement to doc string.

* Fixing flake8 errors

* Cleaning up the tests a smidge.

* A few more flakes

* requested fixes from Andy

* Mini bug fix

* Final pep8 fix

* pep8 fix round 2

* Fix beta_ to alpha_ in the comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants