-
Notifications
You must be signed in to change notification settings - Fork 861
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Adding predict_proba ability to bootstrap_632
functions
#700
Conversation
I'm using a conda env for
|
Thanks for the PR. No worries about the flake8 issues that are not due to new code. They can be fixed another time. Also, we usually don't worry about flake8 issues in /externals, because that's not our code. Both signature_py27.py and six.py can actually be removed because we only support Python 3 now. The error you are getting in the PR is more due to # test predict_proba
> scores = bootstrap_point632_score(lr, X[:100], y[:100],
scoring_func=roc_auc_score,
predict_proba=True,
random_seed=123)
mlxtend/evaluate/tests/test_bootstrap_point632.py:127:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
mlxtend/evaluate/bootstrap_point632.py:185: in bootstrap_point632_score
test_acc = scoring_func(y[test], predict_func(X[test]))
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/utils/validation.py:73: in inner_f
return f(**kwargs)
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:390: in roc_auc_score
return _average_binary_score(partial(_binary_roc_auc_score,
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_base.py:77: in _average_binary_score
return binary_metric(y_true, y_score, sample_weight=sample_weight)
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:226: in _binary_roc_auc_score
fpr, tpr, _ = roc_curve(y_true, y_score,
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/utils/validation.py:73: in inner_f
return f(**kwargs)
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:775: in roc_curve
fps, tps, thresholds = _binary_clf_curve(
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/metrics/_ranking.py:543: in _binary_clf_curve
y_score = column_or_1d(y_score)
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/utils/validation.py:73: in inner_f
return f(**kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
y = array([[0.98158654, 0.01841346],
[0.98338904, 0.01661096],
[0.97811173, 0.02188827],
[0.94485394,...76826, 0.97423174],
[0.00814693, 0.99185307],
[0.05730381, 0.94269619],
[0.02391098, 0.97608902]])
@_deprecate_positional_args
def column_or_1d(y, *, warn=False):
""" Ravel column or 1d numpy array, else raises an error
Parameters
----------
y : array-like
warn : boolean, default False
To control display of warnings.
Returns
-------
y : array
"""
y = np.asarray(y)
shape = np.shape(y)
if len(shape) == 1:
return np.ravel(y)
if len(shape) == 2 and shape[1] == 1:
if warn:
warnings.warn("A column-vector y was passed when a 1d array was"
" expected. Please change the shape of y to "
"(n_samples, ), for example using ravel().",
DataConversionWarning, stacklevel=2)
return np.ravel(y)
> raise ValueError(
"y should be a 1d array, "
"got an array of shape {} instead.".format(shape))
E ValueError: y should be a 1d array, got an array of shape (34, 2) instead.
../../miniconda3/envs/mlxtend-latest/lib/python3.8/site-packages/sklearn/utils/validation.py:846: ValueError |
PS: You can test it efficiently locally via
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for getting it to work! There are two minor comments I have ...
oob = BootstrapOutOfBag(n_splits=n_splits, random_seed=random_seed) | ||
scores = np.empty(dtype=np.float, shape=(n_splits,)) | ||
cnt = 0 | ||
for train, test in oob.split(X): | ||
cloned_est.fit(X[train], y[train]) | ||
|
||
test_acc = scoring_func(y[test], cloned_est.predict(X[test])) | ||
# get the prediction probability | ||
# for binary class uses the last column |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the current implementation support multi-class settings? If not, I think we need the following structure?
if predict_proba:
len_uniq = np.unique(y)
if len(len_uniq) < 2:
# raise error
elif len(len_uniq) == 2:
predicted_train_val = predicted_train_val[:, 1]
predicted_test_val = predicted_test_val[:, 1]
else:
# do something
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't all sklearn
estimators check the degenerate case of a single-class classification? If multiclass, then we can return the full probability array, else in binary classification, get the 2nd column.
I took the general structure though. lmk what you think
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't all sklearn estimators check the degenerate case of a single-class classification?
I think you are right!
Looking at https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html
the multiclass and multilabel cases expect a shape (n_samples, n_classes).
it seems to be okay as it is right now, too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. Will merge. Many thanks!
oob = BootstrapOutOfBag(n_splits=n_splits, random_seed=random_seed) | ||
scores = np.empty(dtype=np.float, shape=(n_splits,)) | ||
cnt = 0 | ||
for train, test in oob.split(X): | ||
cloned_est.fit(X[train], y[train]) | ||
|
||
test_acc = scoring_func(y[test], cloned_est.predict(X[test])) | ||
# get the prediction probability | ||
# for binary class uses the last column |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't all sklearn estimators check the degenerate case of a single-class classification?
I think you are right!
Looking at https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html
the multiclass and multilabel cases expect a shape (n_samples, n_classes).
it seems to be okay as it is right now, too.
Description
Add ability to pass in
scoring_func
tobootstrap_point632_score
function that depends on probability predictions rather then label predictions. This would for example allowroc_auc_score
to be passed into the bootstrapping method.Related issues or pull requests
Closes: #699
Pull Request Checklist
./docs/sources/CHANGELOG.md
file (if applicable)./mlxtend/*/tests
directories (if applicable)mlxtend/docs/sources/
(if applicable)PYTHONPATH='.' pytest ./mlxtend -sv
and make sure that all unit tests pass (for small modifications, it might be sufficient to only run the specific test file, e.g.,PYTHONPATH='.' pytest ./mlxtend/classifier/tests/test_stacking_cv_classifier.py -sv
)flake8 ./mlxtend