Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AlphaSelection not being scored properly #157

Open
bbengfort opened this issue Mar 7, 2017 · 1 comment
Open

AlphaSelection not being scored properly #157

bbengfort opened this issue Mar 7, 2017 · 1 comment
Labels
level: intermediate python coding expertise required priority: medium can wait until after next release type: bug something isn't working
Milestone

Comments

@bbengfort
Copy link
Member

The AlphaSelection visualizer, implemented in #103 has a slight bug:

Right now the alphas and errors selection method uses a search to find the right attributes on the model (rather than responding to specific model names). However, some models return different values for the attributes in different scenarios, resulting in an error regarding the mismatch between x and y values during plotting (must have same shape).

For example, RidgeCV cv_values_ can be:

cv_values_ : array, shape = [n_samples, n_alphas] or shape = [n_samples, n_targets, n_alphas], optional
| Cross-validation values for each alpha (if store_cv_values=True and cv=None). After fit() has been called, this attribute will contain the mean squared errors (by default) or the values of the {loss,score}_func function (if provided in the constructor).

But the current implementation only handles the shape [n_samples, n_alphas].

Additionally ElasticNetCV mse_path_ can be:

mse_path_ : array, shape (n_l1_ratio, n_alpha, n_folds)
| Mean square error for the test set on each fold, varying l1_ratio and
| alpha.

Which means we're probably not doing the right average on this array.

Basically, we need to do a better job of figuring out what the alphas and mse error properties are, computing the scores for visualization; I think right now the plots might just be wrong.

@bbengfort bbengfort added this to the Version 0.3.4 milestone Mar 7, 2017
@bbengfort bbengfort added level: intermediate python coding expertise required priority: medium can wait until after next release type: bug something isn't working labels Mar 7, 2017
@bbengfort
Copy link
Member Author

@NealHumphrey, @balavenkatesan, @ndanielsen -- I could use a second pair of eyes on this if any of you guys had some time to take a look. Checkout my alphas notebook in examples, and the yellowbrick/regressor/alphas.py package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level: intermediate python coding expertise required priority: medium can wait until after next release type: bug something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant