FEA SLEP006: Metadata routing for SelfTrainingClassifier

[adam2392]

Reference Issues/PRs

Towards: #22893

What does this implement/fix? Explain your changes.

Implements metadata routing for SelfTrainingClassifier. The unit-tests need to support scoring, decision_function, and prediction, so I am leaning towards adding this as a unit-test possibly with something like Pipeline?

Or seeing if it can fit inside the existing unit-testing framework w/ the other classifiers like BaggingClassifier.

Any other comments?

cc: @adrinjalali

Some open questions:

1. I presume, we want to forward metadata within all the functions possibly?
2. As a result, I'm not sure if the unit-testing approach is the best, so I was wondering if you have any suggestions? Should I try potentially refactoring the existing unit-testing code to allow testing for more than just fit?

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 523b365. Link to the linter CI: here}

[adrinjalali]

A few notes, thanks @adam2392

[adam2392]

Thank you for the review and pointers! I went thru and fixed the issues in the doc-strings and Bunch.

[adam2392]

Resolved conflicts. Feel free to ping me if there's additional changes desired

[adam2392]

This PR should not be affected by #28734

[adrinjalali]

we can add the other methods here, can't we?

[adam2392]

I can add predict as well. But when I add score and decision_function, I get the following error:

 AssertionError: Expected dict_keys(['sample_weight']) vs dict_keys([])

I already test those other methods technically with the SimpleEstimator class. Idk if that's sufficient? I assumed that Pipeline tests were on their own because of this extra complexity of testing all the methods together, versus within this common test file. Thus the SelfLearningClassifier has a similar pattern as Pipeline

[adrinjalali]

You might need to fix the Consumer object as well for this to work. Pipeline is a different story cause it has several steps and transforming data in between and all, this object is much simpler and it might not need the separate tests at all.

[adam2392]

Okay, I think I got this working now. I reverted the Pipeline SimpleEstimator back and leveraged the entire framework in test_metaestimators_metadata_routing.py instead. The unit tests seem to pass locally.

[adam2392]

Actually I think when I add score to the list of methods to test, I get the following errors. It's challenging for me to figure out why this is occurring tho tbh. Was wondering if you had any pointers?

                # `set_{method}_request({metadata}==True)` on the underlying objects
>               set_requests(
                    estimator,
                    method_mapping=method_mapping,
                    methods=[method_name],
                    metadata_name=key,
                )

sklearn/tests/test_metaestimators_metadata_routing.py:692: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
sklearn/tests/test_metaestimators_metadata_routing.py:548: in set_requests
    set_request_for_method(**{metadata_name: value})
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (), kw = {'metadata': True}

    def func(*args, **kw):
        """Updates the request for provided parameters
    
        This docstring is overwritten below.
        See REQUESTER_DOC for expected functionality
        """
        if not _routing_enabled():
            raise RuntimeError(
                "This method is only available when metadata routing is enabled."
                " You can enable it using"
                " sklearn.set_config(enable_metadata_routing=True)."
            )
    
        if self.validate_keys and (set(kw) - set(self.keys)):
>           raise TypeError(
                f"Unexpected args: {set(kw) - set(self.keys)} in {self.name}. "
                f"Accepted arguments are: {set(self.keys)}"
            )
E           TypeError: Unexpected args: {'metadata'} in score. Accepted arguments are: {'sample_weight'}

sklearn/utils/_metadata_requests.py:1264: TypeError
============================================== short test summary info ===============================================
FAILED sklearn/tests/test_metaestimators_metadata_routing.py::test_error_on_missing_requests_for_sub_estimator[SelfTrainingClassifier]
FAILED sklearn/tests/test_metaestimators_metadata_routing.py::test_setting_request_on_sub_estimator_removes_error[SelfTrainingClassifier]

[adam2392]

Apologies for the delay @adrinjalali. Assuming the CIs are green, then this should be fixed according to your previous comments.

[adam2392]

This try/except pattern makes it very hard to debug the errors and is not explicit. This pattern makes it explicit which methods use y and which do not.

[adam2392]

When score is called, it should unset the method request for score, not fit.