FEA Add Self Training Estimator

[plbecker]

Reference Issues/PRs

Fixes #1243.

What does this implement/fix? Explain your changes.

Implements a meta classifier for semi-supervised learning based on the original Yarowsky self-training algorithm (refer to http://www.aclweb.org/anthology/P95-1026 for details).
Here is a comparison graph of our implemented version on the IRIS dataset. You can find the code under examples/semi_supervised/plot_self_training_performance.py.

Any other comments?

This PR was created in collaboration of @oliverrausch.
This PR is a work in progress and we'd continue working on it if you would be willing to merge once it is completed.

[jnothman]

I think I failed to post an issue, but I've also recently seen work that mentions the effectiveness of tri-training. Do you know anything about the relative effectiveness of these models? I suppose tri-training is harder to define for regression...

[orausch]

For now we've only been testing the simpler, original self-training algorithm which is in this PR, but implementing something similar to tri-training or co-training seems like a natural next step once this feature is done.

[plbecker]

Tri-training looks promising in the paper (outperforming self-training), this might be something we could work on after finishing this PR

[jnothman]

Well, you'd be best pinging in a few weeks, as we are currently trying to push out a release.​

[plbecker]

Sure, that works for us. Is this generally a feature you would potentially include (once it is done)?

[jnothman]

Is this generally a feature you would potentially include?

I think it's an area where we should improve our current offerings, yes. ​

[jnothman]

All the CI say no! :\

[plbecker]

Yes, we will fix that as soon as possible. We have exams for the next two weeks and will get back to it then.

[jnothman]

Why are these skipped?

[orausch]

I'm skipping transform and inverse_transform because, to my understanding, they aren't part of the estimator API (and therefore I don't expose them as methods).

predict_proba is skipped because the SelfTrainingClassifier throws an error if it is missing.

predict_log_proba and predict are no longer skipped.

[orausch]

Concerning score, I think there might be inconsistency in the test. The score method I use comes straight from ClassifierMixin: with the following signature
def score(self, X, y, sample_weight=None):

However the test calls score with only X, which causes it to fail. I suspect that the reason for this is that since Pipeline allows score to be called with y=None, the test uses the following score method for the estimator:
def score(self, X, *args, **kwargs):

This causes the test to fail on my classifier because ClassifierMixin expects a y argument.

[orausch]

In general, I'm a bit confused as to what the score method is supposed to do, since some classifiers (for example ClassifierMixin, RFE, EllipticEnvelope) return the accuracy score on the passed X and y.

However, other estimators, like PCA and KernelDensity use their own version of score that completely ignores y and only uses X.

[jnothman]

Yes, score can be a little confusing. But PCA and KernelDensity are not supervised so their evaluation is not by default against a ground truth. They report model likelihoods instead.

Yes, it might be a mistake in the test to not try passing y to score.

[orausch]

I've made a PR to fix the test. Once it is merged I will be able to remove score from the skipped methods.

[orausch]

@jnothman Thanks for the review, I'll be addressing those issues shortly. I have a question about checking if SelfTrainingClassifier is fitted. I know about sklearn.utils.validation.check_is_fitted, but it only seems to verify that an attribute exists.

Since this class (unlike other estimators) doesn't create any attributes during the fitting process that would signify that fitting is done, would it be acceptable to introduce a signifier variable (something like is_fitted) during the fit method, such that check_is_fitted(self, "is_fitted") returns true iff the estimator is fitted?

EDIT: or I could perhaps create an attribute base_estimator_ that is cloned from base_estimator at the start of the fit method, and then fit that estimator.

[jnothman]

Can't you just delegate to the underlying estimator to check if fitted?

[orausch]

Yes, but I decided against this because this would erroneously report the estimator as fitted if a fitted estimator is passed to the class. This seems like incorrect behavior to me.

I will be adding the predicted labels and the iteration they were labeled as attributes, so I'll check if the estimator is fitted based on those.

[lorentzenchr]

@orausch Thanks for addressing my comments.

Following the conversion of this PR, several core developers seemed to welcome this as a good addition, but none approved. Currently, I can only assess the code which is in a good shape, only very little work left. I cannot, however, judge the usefulness, robustness and safety of this self training semi-supervised algorithm. A little advocate in my head is asking: How can self training gain more information about the relation between label y and features X than there is in the labeled data alone? I'd like see see an example demonstrating exactly this: a performance comparison of a classifier fit on labeled data alone with self-trained classifier using labeled + unlabeled data.

Therefore my question: Shall we postpone and milestone v0.25 instead? This would give us time to find and evaluate such an example.

[orausch]

How can self training gain more information about the relation between label y and features X than there is in the labeled data alone?

If this wasn't possible semi-supervised learning wouldn't exist as a term 😄. Joking aside, I anecdotally would say that self-training is one of the more established/popular approaches to semi-supervised learning. What kind of "example" would you have in mind? We have a toy example in this PR, but it's understandably not convincing.

I think it might be good to ping some of the previous reviewers to get their thoughts on this @jnothman @NicolasHug.

Interesting side note: I just checked and the linked issue is the 20th oldest in sklearn!

[lorentzenchr]

If this wasn't possible semi-supervised learning wouldn't exist as a term 😄.

I come from a very supervised environment 🤣 (inside joke)

[lorentzenchr]

It would be nice to have one rbf_svc_30, fitted on y_30.

Can you also print the achieved accuracy of all the models on the full (or better test) dataset?

[orausch]

Hmm, I'm not really sure how we could do that without making the layout messy.

[rth]

We have a toy example in this PR, but it's understandably not convincing.

Yes, I think finding a convincing example is the main blocker from merging this PR.
From feedback I heard while discussing this PR with other core devs a few months ago, for instance label propagation, another semi-supervised model merged some time ago isn't working very well in practical applications, and we would like to avoid that situation here.

[ogrisel]

Indeed, semi-supervised learning can probably only really shine on medium sized dataset with at least a few hundreds labels samples and at least a few thousand unlabeled samples.

Maybe trying to demonstrate its usefulness on 20 newsgroups classification would work? This has 18k samples. I assume we could randomly extract 10 to 100 samples per class in the training set and consider all the remaining training set samples as unlabeled. For the model we could use a pipeline of TfIdfVectorizer with ngram_range=(1, 2), min_df=5 and max_df=0.8 + SGDClassifier with some amount l2 or elasticnet reguarlization.

Feel free to take inspiration from https://scikit-learn.org/stable/auto_examples/model_selection/grid_search_text_feature_extraction.html#sphx-glr-auto-examples-model-selection-grid-search-text-feature-extraction-py to design the pipeline.

Alternatively, one might consider the adult census dataset from openml.

For the result it would be interesting to compare to test accuracy the self-training classifier trained on 10% of the training samples labeled (+ the remaining samples considered as unlabeled) vs the 100% supervised equivalent model.

[ogrisel]

I removed the milestone to not block the release but if a convincing example is contributed before the final release we can still consider merging this PR in time.

[orausch]

@ogrisel I wrote up a quick example for the newsgroups dataset: https://gist.github.com/orausch/acf62e3fbf0ea5176f768d3dd3340ae1

Would something like this be sufficient?

[ogrisel]

Would something like this be sufficient?

This looks great indeed. Please open a PR add it to the PR.

[rth]

Thanks @orausch ! I did another pass on the code, generally it LGTM. The example looks quite good as well.

A few minor comments are below, and it would be good to mention somewhere (in the parameter docstring) that the value of the threshold is linked to the calibration of the classifier, ideally with a link to the calibration user-guide section.

[rth]

I think we can just run the example (rename it to plot_*), and remove the example of output above? If the run time is an issue, as far as I can tell due to LabelSpreading, we can skip it in CI:

if 'CI' not in os.environ:
   # LabelSpreading takes too long to run in the online documentation
   # add code here

Without LabelSpreading it should run in under in less than 30s I think?

The 20 newsgroup dataset is already used in other examples so it should be in the cache.

[cmarmo]

One approval... almost there... :)

[lorentzenchr]

The new example on the 20 newsgroup shows the added value of self-training, thanks!

[lorentzenchr]

@orausch @plbecker Thanks for this new meta-estimator.
@orausch Thank you very much for you effort latetly. Let's squeeze this into the next upcoming release🚀

[lorentzenchr]

CI failure in "Test Docs" seems unrelated. CI has been completely green before my last change. Therefore merging.

=================================== FAILURES ===================================
_____________________ [doctest] loading_other_datasets.rst _____________________
093 data and experiments, that allows everybody to upload open datasets.
094 
095 The ``sklearn.datasets`` package is able to download datasets
096 from the repository using the function
097 :func:`sklearn.datasets.fetch_openml`.
098 
099 For example, to download a dataset of gene expressions in mice brains::
100 
101   >>> from sklearn.datasets import fetch_openml
102   >>> mice = fetch_openml(name='miceprotein', version=4)
UNEXPECTED EXCEPTION: OpenMLError('Dataset with data_id 40966 not found.')
Traceback (most recent call last):
  File "/usr/share/miniconda/envs/testvenv/lib/python3.9/doctest.py", line 1336, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest loading_other_datasets.rst[7]>", line 1, in <module>
  File "/home/vsts/work/1/s/sklearn/utils/validation.py", line 60, in inner_f
    return f(*args, **kwargs)
  File "/home/vsts/work/1/s/sklearn/datasets/_openml.py", line 873, in fetch_openml
    features_list = _get_data_features(data_id, data_home)
  File "/home/vsts/work/1/s/sklearn/datasets/_openml.py", line 450, in _get_data_features
    json_data = _get_json_content_from_openml_api(
  File "/home/vsts/work/1/s/sklearn/datasets/_openml.py", line 179, in _get_json_content_from_openml_api
    raise OpenMLError(error_message)
sklearn.datasets._openml.OpenMLError: Dataset with data_id 40966 not found.
/home/vsts/work/1/s/doc/datasets/loading_other_datasets.rst:102: UnexpectedException