Alphas Plot

[rebeccabilbro]

Create some new type of visualizer that enables the user to plot the error for different alphas. Line graph - use the line to show which alpha was selected.

[bbengfort]

I've started investigating this issue this morning; it seems like a fairly straight forward implementation. For reference, here is the implementation from the Machine Learning slides. However, I believe the best way to implement this visualizer is to wrap one of the RegressorCV models, e.g. RidgeCV or LassoCV, etc.

This is the manual method:

from sklearn.linear_model import Ridge
from sklearn.cross_validation import cross_val_score
from sklearn.metrics import mean_squared_error

errors = []
for alpha in alphas:
    model = Ridge(alpha=alpha)
    score = cross_val_score(model, X, y, cv=12, scoring='mean_squared_error')
    errors.append(score.mean())

plt.plot(alphas, errors, label='ridge')

And here is the result:

So obviously something went weird there. Also I was expecting a more jagged result, more like:

So there are interesting things to look at here!

[bbengfort]

More investigation reveals that RidgeCV, LassoCV, ElasticNetCV, and LassoLarsCV all have different attributes to fetch for their models:

RidgeCV:

from sklearn.linear_model import RidgeCV 

# Must pass store_cv_values=True in order to store the MSE of the alphas. 
ridge = RidgeCV(alphas=alphas, store_cv_values=True)
ridge.fit(X, y)

# Plot against ridge.alphas and ridge.cv_values_
g = plt.plot(ridge.alphas, ridge.cv_values_.mean(0), label="ridge")

LassoCV:

from sklearn.linear_model import LassoCV 

lasso = LassoCV(alphas=alphas)
lasso.fit(X, y)

# plot against lasso.alphas_ and lasso.mse_path_
g = plt.plot(lasso.alphas_, lasso.mse_path_.mean(1), label="lasso")

[bbengfort]

ElasticNetCV:

from sklearn.linear_model import ElasticNetCV 

enet = ElasticNetCV(alphas=alphas)
enet.fit(X, y)

g = plt.plot(enet.alphas_, enet.mse_path_.mean(1), label="elasticnet")

LassoLarsCV:

from sklearn.linear_model import LassoLarsCV 

larslasso = LassoLarsCV()
larslasso.fit(X, y)

# plot against lars.cv_alphas_ and lars.cv_mse_path_
g = plt.plot(larslasso.cv_alphas_, larslasso.cv_mse_path_.mean(1), label="lasso lars")

All of them on the same graph:

[bbengfort]

Note: we should also put the selected alpha and final cross validation score into the plot as well!

[bbengfort]

AlphaSelection visualizer implemented, as well as a refactoring of the regressor package!

Here are some screenshots:

from yellowbrick.regressor import AlphaSelection 
from sklearn.linear_model import RidgeCV, LassoCV, LassoLarsCV, ElasticNetCV

alphas = np.logspace(-12, -1, 400)

model = AlphaSelection(RidgeCV(alphas=alphas, scoring="neg_mean_squared_error"))
model.fit(X, y)
model.poof()

model = AlphaSelection(LassoCV(alphas=alphas))
model.fit(X, y)
model.poof()

model = AlphaSelection(LassoLarsCV())
model.fit(X, y)
model.poof()

model = AlphaSelection(ElasticNetCV(alphas=alphas))
model.fit(X, y)
model.poof()

[bbengfort]

Done but not in examples; closing this issue and filing bug report for the examples thing.

[BradKML]

Thanks for the note, but I am also encountering the LassoLarsCV issue as they don't require someone to define alphas.
Also as a side note how can LassoLarsIC use this?