[WIP] Adding predict_proba ability to bootstrap_632 functions
#700
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I'm using a conda env for |
|
Thanks for the PR. No worries about the flake8 issues that are not due to new code. They can be fixed another time. Also, we usually don't worry about flake8 issues in /externals, because that's not our code. Both signature_py27.py and six.py can actually be removed because we only support Python 3 now. The error you are getting in the PR is more due to # test predict_proba
> scores = bootstrap_point632_score(lr, X[:100], y[:100],
scoring_func=roc_auc_score,
predict_proba=True,
random_seed=123)
mlxtend/evaluate/tests/test_bootstrap_point632.py:127:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mlxtend/evaluate/bootstrap_point632.py:185: in bootstrap_point632_score
test_acc = scoring_func(y[test], predict_func(X[test]))
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/utils/validation.py:73: in inner_f
return f(**kwargs)
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:390: in roc_auc_score
return _average_binary_score(partial(_binary_roc_auc_score,
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_base.py:77: in _average_binary_score
return binary_metric(y_true, y_score, sample_weight=sample_weight)
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:226: in _binary_roc_auc_score
fpr, tpr, _ = roc_curve(y_true, y_score,
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/utils/validation.py:73: in inner_f
return f(**kwargs)
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:775: in roc_curve
fps, tps, thresholds = _binary_clf_curve(
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:543: in _binary_clf_curve
y_score = column_or_1d(y_score)
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/utils/validation.py:73: in inner_f
return f(**kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
y = array([[0.98158654, 0.01841346],
[0.98338904, 0.01661096],
[0.97811173, 0.02188827],
[0.94485394,...76826, 0.97423174],
[0.00814693, 0.99185307],
[0.05730381, 0.94269619],
[0.02391098, 0.97608902]])
@_deprecate_positional_args
def column_or_1d(y, *, warn=False):
""" Ravel column or 1d numpy array, else raises an error
Parameters
----------
y : array-like
warn : boolean, default False
To control display of warnings.
Returns
-------
y : array
"""
y = np.asarray(y)
shape = np.shape(y)
if len(shape) == 1:
return np.ravel(y)
if len(shape) == 2 and shape[1] == 1:
if warn:
warnings.warn("A column-vector y was passed when a 1d array was"
" expected. Please change the shape of y to "
"(n_samples, ), for example using ravel().",
DataConversionWarning, stacklevel=2)
return np.ravel(y)
> raise ValueError(
"y should be a 1d array, "
"got an array of shape {} instead.".format(shape))
E ValueError: y should be a 1d array, got an array of shape (34, 2) instead.
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/utils/validation.py:846: ValueError |
|
PS: You can test it efficiently locally via |
rasbt
reviewed
Jul 1, 2020
| oob = BootstrapOutOfBag(n_splits=n_splits, random_seed=random_seed) | ||
| scores = np.empty(dtype=np.float, shape=(n_splits,)) | ||
| cnt = 0 | ||
| for train, test in oob.split(X): | ||
| cloned_est.fit(X[train], y[train]) | ||
|
|
||
| test_acc = scoring_func(y[test], cloned_est.predict(X[test])) | ||
| # get the prediction probability | ||
| # for binary class uses the last column |
There was a problem hiding this comment.
Does the current implementation support multi-class settings? If not, I think we need the following structure?
if predict_proba:
len_uniq = np.unique(y)
if len(len_uniq) < 2:
# raise error
elif len(len_uniq) == 2:
predicted_train_val = predicted_train_val[:, 1]
predicted_test_val = predicted_test_val[:, 1]
else:
# do somethingThere was a problem hiding this comment.
Don't all sklearn estimators check the degenerate case of a single-class classification? If multiclass, then we can return the full probability array, else in binary classification, get the 2nd column.
I took the general structure though. lmk what you think
There was a problem hiding this comment.
Don't all sklearn estimators check the degenerate case of a single-class classification?
I think you are right!
Looking at https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html
the multiclass and multilabel cases expect a shape (n_samples, n_classes).
it seems to be okay as it is right now, too.
rasbt
approved these changes
Jul 3, 2020
| oob = BootstrapOutOfBag(n_splits=n_splits, random_seed=random_seed) | ||
| scores = np.empty(dtype=np.float, shape=(n_splits,)) | ||
| cnt = 0 | ||
| for train, test in oob.split(X): | ||
| cloned_est.fit(X[train], y[train]) | ||
|
|
||
| test_acc = scoring_func(y[test], cloned_est.predict(X[test])) | ||
| # get the prediction probability | ||
| # for binary class uses the last column |
There was a problem hiding this comment.
Don't all sklearn estimators check the degenerate case of a single-class classification?
I think you are right!
Looking at https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html
the multiclass and multilabel cases expect a shape (n_samples, n_classes).
it seems to be okay as it is right now, too.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add ability to pass in
scoring_functobootstrap_point632_scorefunction that depends on probability predictions rather then label predictions. This would for example allowroc_auc_scoreto be passed into the bootstrapping method.Related issues or pull requests
Closes: #699
Pull Request Checklist
./docs/sources/CHANGELOG.mdfile (if applicable)./mlxtend/*/testsdirectories (if applicable)mlxtend/docs/sources/(if applicable)PYTHONPATH='.' pytest ./mlxtend -svand make sure that all unit tests pass (for small modifications, it might be sufficient to only run the specific test file, e.g.,PYTHONPATH='.' pytest ./mlxtend/classifier/tests/test_stacking_cv_classifier.py -sv)flake8 ./mlxtend