Implement independent predict and score methods for all estimators

[grovduck]

The IndependentPredictionMixin class provides two methods (predict_independent and score_independent) that don't include the query point itself when finding nearest neighbors. These methods are similar in functionality to calling KNeighborsRegressor.kneighbors() without passing X. In this way, only "independent" neighbors are used in calculating predictions and scores.

EDIT: This PR evolved into addressing some class hierarchy issues with our current mixins and inheritance from scikit-learn's KNeighborsRegressor. Read through the PR for details on what has changed.

This PR will fix #45.

[grovduck]

@aazuspan, definitely in draft state pending your ideas. Lots of drama for what amounts to five lines of code to make the independent predict and score work correctly 😉.

[aazuspan]

Looks great @grovduck!

I mentioned in the issue that I slightly prefer the fit attributes to independent methods, but seeing this in practice, I could go either way! Maybe the question should be whether you'd ever call score_independent with a different y than was used to fit the model? If so, a method definitely makes more sense.

Lots of drama for what amounts to five lines of code to make the independent predict and score work correctly 😉.

Nothing's ever as simple as it should be!

[grovduck]

I mentioned in the issue that I slightly prefer the fit attributes to independent methods, but seeing this in practice, I could go either way! Maybe the question should be whether you'd ever call score_independent with a different y than was used to fit the model? If so, a method definitely makes more sense.

@aazuspan, I can't think of an instance where we'd need to call score_independent with a different y, so I think the fit attributes approach should work and I agree is preferable. It seems like the natural place for this would be in TransformedKNeighborsMixin.fit like this?

class TransformedKNeighborsMixin:
    def fit(self, X, y):
        """Fit using transformed feature data."""
        X_transformed = self._apply_transform(X)
        self = super().fit(X_transformed, y)
        self.independent_prediction_ = super().predict(X=None)
        self.independent_score_ = super().score(X=None, y=y)
        return self

Is this what you were thinking? I think we'd have to introduce a similar fit method for RawKNNRegressor as it doesn't derive from TransformedKNeighborsMixin. We'd obviously also be able to get rid of IndependentPredictionMixin.

[aazuspan]

Is this what you were thinking? I think we'd have to introduce a similar fit method for RawKNNRegressor as it doesn't derive from TransformedKNeighborsMixin.

That was what I was thinking, but having to duplicate it for RawKNNRegressor definitely isn't ideal, and I think is indicative of the fact that it probably shouldn't be tied to the transformed mixin since it's not really relevant whether the estimator has a transformer.

Alternatively, we could 1) keep IndependentPredictionMixin and override fit to add those attributes, or 2) add it to KNeighborsDFIndexCrosswalkMixin.fit since every estimator already inherits from that, and maybe give the mixin a more general name. I don't love the complexity of adding another fit method to the MRO with option 1, but I'm also not sure about mixing different functionality in a single mixin with option 2...

Any thoughts on those options or other ideas?

[grovduck]

Alternatively, we could 1) keep IndependentPredictionMixin and override fit to add those attributes, or 2) add it to KNeighborsDFIndexCrosswalkMixin.fit since every estimator already inherits from that, and maybe give the mixin a more general name. I don't love the complexity of adding another fit method to the MRO with option 1, but I'm also not sure about mixing different functionality in a single mixin with option 2...

Any thoughts on those options or other ideas?

Yep, good call! Option 2 makes me uneasy partly because of the mixed functionality that you suggest (and partly because I can't come up with a succinct name that would incorporate both sets of functionality!).

The only other option that I can think of (that doesn't really add anything) is to directly subclass KNeighborsRegressor into a new derived class, implement this functionality there in fit, and have all the other estimators derive from it. (My only reason for suggesting this one is that this functionality seems like it could generically be added to KNeighborsRegressor to mirror kneighbors without issue. But unless sklearn decides to take it on themselves, we've got to deal with it!)

So absent those solutions, I think my default is to keep it in the IndependentPredictionMixin class. It makes it clear what the mixin is doing even if it does clutter the MRO. I swear, it will be the last one!

[aazuspan]

The only other option that I can think of (that doesn't really add anything) is to directly subclass KNeighborsRegressor into a new derived class, implement this functionality there in fit, and have all the other estimators derive from it.

I like this suggestion! For one thing, it will simplify the inheritance a little bit, and for another I've been starting to feel uneasy about the fact that we have mixins that all of our estimators are using, which seems to sort of defeat the composability benefit of mixins. I haven't fully thought through the MRO and whether everything will get called in the correct order, but if that all works out then this has my vote.

Unfortunately, I'm going to throw a wrench in the works by asking whether we should also combine KNeighborsDFIndexCrosswalkMixin into this base class? My thinking again is that every estimator uses that dataframe functionality, so keeping it in a separate mixin feels a little odd and unnecessarily complex. So we'd essentially be building a subclass of KNeighborsRegressor where we could put all of the special features that all our estimators can use. This still mixes functionality, but for some reason I feel better about it if it's a base class instead of a mixin (although I still have no idea what we would call it!). Feel free to veto the idea, or we could backburner it and just focus on getting the new functionality in for now.

[grovduck]

Unfortunately, I'm going to throw a wrench in the works by asking whether we should also combine KNeighborsDFIndexCrosswalkMixin into this base class?

Totally a fair question. I'll run with this and get KNeighborsDFIndexCrosswalkMixin in the subclass as well. It feels better to me as well to have that mixed functionality in a subclass rather than a mixin. I'm going to push a temporary commit that uses the mixin solution (along with your other suggested changes), just so we have a commit to revert to if everything goes haywire.

As far as naming this subclass, what do you think of making it a "private" class (e.g. _KNeighborsRegressor)? This would suggest to users that they shouldn't use the class directly, but that it will be very similar to sklearn's version. I don't have any idea if there is precedent for this or reasons why this is definitely not the way to go.

[aazuspan]

As far as naming this subclass, what do you think of making it a "private" class (e.g. _KNeighborsRegressor)? This would suggest to users that they shouldn't use the class directly, but that it will be very similar to sklearn's version. I don't have any idea if there is precedent for this or reasons why this is definitely not the way to go.

I think I like that idea! Using an abstract base class would be another way to make it clear that it shouldn't be instantiated directly, but that has some other implications that I don't think we want to use, like requiring subclasses to implement their own versions of methods. Going with a private class seems like a good straightforward solution (and avoids the challenge of coming up with a unique name!).

[grovduck]

I haven't fully thought through the MRO and whether everything will get called in the correct order

I think we're bumping into some MRO issues as you thought might happen. Here's a draft _KNeighborsRegressor which is a simple combine of KNeighborsDFIndexCrosswalkMixin and IndependentPredictionMixin:

class _KNeighborsRegressor(KNeighborsRegressor):
    """
    Subclass of `sklearn.neighbors.KNeighborsRegressor` to support independent
    prediction and scoring and crosswalk array indices to dataframe indexes.

    This class serves as the superclass for many estimators in this package, but
    should not be instantiated directly.
    """

    def fit(self, X, y):
        """Store dataframe indexes if X is a dataframe and store fit attributes."""
        if hasattr(X, "index"):
            self.dataframe_index_in_ = np.asarray(X.index)

        self = super().fit(X, y)
        self.independent_prediction_ = super().predict(X=None)
        self.independent_score_ = super().score(X=None, y=y)
        return self

    def kneighbors(
        self,
        X=None,
        n_neighbors=None,
        return_distance=True,
        return_dataframe_index=False,
    ):
        """Override kneighbors to optionally return dataframe indexes."""
        neigh_dist, neigh_ind = super().kneighbors(
            X=X, n_neighbors=n_neighbors, return_distance=True
        )

        if return_dataframe_index:
            msg = "Dataframe indexes can only be returned when fitted with a dataframe."
            check_is_fitted(self, "dataframe_index_in_", msg=msg)
            neigh_ind = self.dataframe_index_in_[neigh_ind]

        return (neigh_dist, neigh_ind) if return_distance else neigh_ind

For the estimators, we can choose the order of the superclasses, but let's assume that we list TransformedKNeighborsMixin before _KNeighborsRegressor. EuclideanKNNRegressor, for example, becomes:

class EuclideanKNNRegressor(
    TransformedKNeighborsMixin,
    _KNeighborsRegressor
):
    def fit(self, X, y):
        self.transform_ = StandardScalerWithDOF(ddof=1).fit(X)
        return super().fit(X, y)

When we call fit, it first hits TransformedKNeighborsMixin.fit, applies the transform and then calls _KNeighborsRegressor.fit. At this point, X is an array, so it no longer has an attribute called "index" if we had originally passed a dataframe. Because KNeighborsDFIndexCrosswalkMixin was first in the class order before, we didn't have this issue.

If we reverse the order of the superclases in the estimators (e.g. (_KNeighborsRegressor, TransformedKNeighborsMixin)), _KNeighborsRegressor.fit gets called first, but because it directly subclasses KNeighborsRegressor, that fit is called next before TransformedKNeighborsMixin.fit. This is causing test failures as well.

I'm hoping you have a typical brilliant solution!!

[aazuspan]

This is a tricky one! I definitely don't have any quick fixes coming to mind, so in the interest of trying to think through the problem out loud...

I would say that TransformedKNeighborsMixin should be inherited before the _KNeighborsRegressor base class, but that would mean that we need to preserve the dataframe index (and maybe other attributes in the future) through the transformation. Obviously we tried that in the past with the NDArray subclass to store feature names and with using the set_output API to return dataframes, and neither one was a very elegant solution. Alternatively, TransformedKNeighborsMixin.fit could store dataframe attributes like index as fitted attributes prior to transformation, but that would mean _KNeighborsRegressor.fit would need to check for both self.index_ and X.index when setting dataframe_index_in_, to account for the untransformed Raw regressor...

There may be a really clever solution for reworking our inheritance that solves all these problems, but I think I'm stumped at the moment. Assuming you haven't had any breakthroughs on this, I'm happy to go with the IndependentPredictionMixin solution for now and track the mixin refactor as a separate issue, if you want to keep this moving. Your call!

[grovduck]

@aazuspan, I'll throw this one out there, but I think I'm going to call you the "master architect" here, so you can tell me if you find this distasteful. Instead of using TransformedKNeighborsMixin as a mixin, we could instead treat it as a subclass of _KNeighborsRegressor, so then we would have a strictly vertical inheritance:

`sklearn.KNeighborsRegressor <- _KNeighborsRegressor <- TransformedKNeighborsRegressor`

So then EUC, MAH, MSN, and GNN all have TransformedKNeighborsRegressor as their superclass and RAW still uses _KNeighborsRegressor as its superclass. I played around with this and it does pass all tests with the following changes:

class _KNeighborsRegressor(KNeighborsRegressor):
    """
    Subclass of `sklearn.neighbors.KNeighborsRegressor` to support independent
    prediction and scoring and crosswalk array indices to dataframe indexes.

    This class serves as the superclass for many estimators in this package, but
    should not be instantiated directly.
    """
    # This method is now split out and called from TransformedKNeighborsRegressor.fit
    # and RawKnnRegressor.fit
    def _set_dataframe_index_in(self, X):
        """Store dataframe indexes if X is a dataframe."""
        if hasattr(X, "index"):
            self.dataframe_index_in_ = np.asarray(X.index)

    # Same as before without the setting of the dataframe index
    def fit(self, X, y):
        """Store dataframe indexes if X is a dataframe and store fit attributes."""
        self = super().fit(X, y)
        self.independent_prediction_ = super().predict(X=None)
        self.independent_score_ = super().score(X=None, y=y)
        return self

    # Same as original KNeighborsDFIndexCrosswalkMixin.kneighbors
    def kneighbors(
        self,
        X=None,
        n_neighbors=None,
        return_distance=True,
        return_dataframe_index=False,
    ):
        """Override kneighbors to optionally return dataframe indexes."""
        neigh_dist, neigh_ind = super().kneighbors(
            X=X, n_neighbors=n_neighbors, return_distance=True
        )

        if return_dataframe_index:
            msg = "Dataframe indexes can only be returned when fitted with a dataframe."
            check_is_fitted(self, "dataframe_index_in_", msg=msg)
            neigh_ind = self.dataframe_index_in_[neigh_ind]

        return (neigh_dist, neigh_ind) if return_distance else neigh_ind


# Changed name from Mixin to Regresssor
class TransformedKNeighborsRegressor(_KNeighborsRegressor):
    """
    Class for KNeighbors regressors that apply transformations to the feature data.
    """

    transform_: TransformerMixin

    @property
    def feature_names_in_(self):
        return self.transform_.feature_names_in_

    @property
    def n_features_in_(self):
        return self.transform_.n_features_in_

    def _check_feature_names(self, X, *, reset):
        """Override BaseEstimator._check_feature_names to prevent errors.

        This check would fail during fitting because `feature_names_in_` stores original
        names while X contains transformed names. We instead rely on the transformer to
        check feature names and warn or raise for mismatches.
        """
        return

    def _check_n_features(self, X, *, reset):
        """Override BaseEstimator._check_n_features to prevent errors.

        See _check_feature_names.
        """
        return

    def _apply_transform(self, X) -> NDArray:
        """Apply the stored transform to the input data."""
        check_is_fitted(self, "transform_")
        return self.transform_.transform(X)

    # Same as before but calls _set_dataframe_index_in_
    def fit(self, X, y):
        """Fit using transformed feature data."""
        self._set_dataframe_index_in(X)
        X_transformed = self._apply_transform(X)
        return super().fit(X_transformed, y)

    # Same as before, but adds a **kwargs parameter so that function calls with
    # return_dataframe_index can pass through
    def kneighbors(self, X=None, n_neighbors=None, return_distance=True, **kwargs):
        """Return neighbor indices and distances using transformed feature data."""
        X_transformed = self._apply_transform(X) if X is not None else X
        return super().kneighbors(
            X=X_transformed,
            n_neighbors=n_neighbors,
            return_distance=return_distance,
            **kwargs,
        )

Thoughts? Too deep of a class hierarchy? I know it goes against the mix-in philosophy of sklearn a bit, so that might be reason to not use it. If you think this is too much of a change, I think I'll go back to the IndependentPredictionMixin design for now and kick this down the road.

[aazuspan]

I'm totally open to this design! It does seem less idiomatic to the sklearn style, but ultimately probably more "Pythonic". Even though we still technically have to set dataframe_index_in_ from the TransformedKNeighbors... subclass, being able to inherit the method from the base class feels a lot better. Some questions and comments below, but overall I think I'm in favor of this approach!

1. Trying to look ahead, do you think this would have any negative implications for RF-NN or any other possible future estimators? I don't see why it would, but wanted to check since I think you have a pretty good idea of what that will look like already.

2. How does RawKNNRegressor (and potentially other non-transformed estimators) set dataframe_index_in_? Would we call that from _KNeighborsRegressor.fit with the idea that any transformed estimator already has it set and will be passing an array anyways? Or do we implement RawKNNRegressor.fit and call it from there? I would lean towards the former since it is a feature of _KNeighborsRegressor, even though that means it gets called twice for transformed estimators. What do you think?

3. Given that TransformedKNeighborsRegressor is now another class that shouldn't be instantiated directly, do we make that private as well? I would think either that or we should come up with another name for _KNeighborsRegressor, since the inconsistency feels a little odd.

4. I suggest we make return_dataframe_index an explicit argument for TransformedKNeighborsRegressor.kneighbors, for the sake of docstrings and IDE autocompletion.

[grovduck]

I'm totally open to this design! It does seem less idiomatic to the sklearn style, but ultimately probably more "Pythonic". Even though we still technically have to set dataframe_index_in_ from the TransformedKNeighbors... subclass, being able to inherit the method from the base class feels a lot better. Some questions and comments below, but overall I think I'm in favor of this approach!

Excellent! I'll go ahead and create a commit for your review. I've made other biggish changes below and thought it would be easier to just get a commit in there.

1. Trying to look ahead, do you think this would have any negative implications for RF-NN or any other possible future estimators? I don't see why it would, but wanted to check since I think you have a pretty good idea of what that will look like already.

This is a great question! RF-NN won't be able to use KNeighborsRegressor (different way of finding distance), but probably would benefit from functionality in (what was) KNeighborsDFIndexCrosswalkMixin and IndependentPredictionMixin. I think it's possible to do this (yet another class design change):

DFIndexCrosswalkMixin <----------
                                |
IndependentPredictorMixin <-----|---- _KNeighborsRegressor <----- _TransformedKNeighborsRegressor
                                |
KNeighborsRegressor <------------

I think I really like this design because DFIndexCrosswalkMixin is really kNN independent and IndependentPredictorMixin should be able to apply to other estimators where you want to do an "independent" prediction and score from the X data passed to fit. So essentially it frees these mixins from being specifically kNN. (EDIT: As @aazuspan notes in the next post, DFIndexCrosswalkMixin still relies on _KNeighborsRegressor to do the actual crosswalking of neighbor's dataframe index to the array index, so this mixin, while still independent, relies on the subclass to implement any crosswalking).

2. How does RawKNNRegressor (and potentially other non-transformed estimators) set dataframe_index_in_? Would we call that from _KNeighborsRegressor.fit with the idea that any transformed estimator already has it set and will be passing an array anyways? Or do we implement RawKNNRegressor.fit and call it from there? I would lean towards the former since it is a feature of _KNeighborsRegressor, even though that means it gets called twice for transformed estimators. What do you think?

Yep, initially I had RawKnnRegressor implement fit but I agree that putting it in _KNeighborsRegressor is a better match even if it's called twice.

3. Given that TransformedKNeighborsRegressor is now another class that shouldn't be instantiated directly, do we make that private as well? I would think either that or we should come up with another name for _KNeighborsRegressor, since the inconsistency feels a little odd.

I totally agree. I took the lazy way out and have renamed to _TransformedKNeighborsRegressor. I'm not sure I think this is awesome. Please change if you have inspiration.

4. I suggest we make return_dataframe_index an explicit argument for TransformedKNeighborsRegressor.kneighbors, for the sake of docstrings and IDE autocompletion.

Yep, I agree here as well. Implemented in the commit.

[aazuspan]

I think I really like this design because DFIndexMixin is really kNN independent and IndependentPredictorMixin should be able to apply to other estimators where you want to do an "independent" prediction and score from the X data passed to fit. So essentially it frees these mixins from being specifically kNN.

This is great! I think you're 100% right about splitting out the kNN-independent functionality, and this is a really clean and clear way to go about it. I do feel a little weird that DFIndexCrosswalkMixin currently just sets an attribute, but obviously we can't put the actual crosswalking functionality there without tying it to kNN, so this seems like a necessary compromise.

Overall, I think this is a big improvement to readability and gives us a lot more flexibility going forward. Great work!

I took the lazy way out and have renamed to _TransformedKNeighborsRegressor. I'm not sure I think this is awesome. Please change if you have inspiration.

I'm happy with this, but I do have an alternative idea for us to consider. I think we discussed earlier that it's a little odd that RawKNNRegressor is just an alias for what is now _KNeighborsRegressor, but we wanted to expose that class in the public API somehow. What do you think about renaming _KneighborsRegressor to RawKNNRegressor? So conceptually, something like EuclideanKNNRegressor would effectively be a RawKNNRegressor with the addition of a transformer. Does that make sense to you, or would you rather keep the separation between the base classes and the estimators?

[grovduck]

I do feel a little weird that DFIndexCrosswalkMixin currently just sets an attribute, but obviously we can't put the actual crosswalking functionality there without tying it to kNN, so this seems like a necessary compromise.

Yes, you're absolutely right about this. I've edited my comment above to reflect this. This doesn't preclude other classes that use this mixin from implement their own crosswalk, but I agree that it may not have a lot of utility outside of kNN estimators. Still, as you note, I think it's a compromise we can live with.

I think we discussed earlier that it's a little odd that RawKNNRegressor is just an alias for what is now _KNeighborsRegressor, but we wanted to expose that class in the public API somehow. What do you think about renaming _KneighborsRegressor to RawKNNRegressor? So conceptually, something like EuclideanKNNRegressor would effectively be a RawKNNRegressor with the addition of a transformer.

Perfect! Great idea and I think that's an intuitive way to handle this. I've pushed another commit with this change. I decided to remove _raw.py and just rename _KNeighborsRegressor to RawKNNRegressor in _base.py. I think we were going to get caught in a circular dependency otherwise (RawKNNRegressor would rely on the mixins in _base, and then _TransformedKNeighborsRegressor would rely on _raw). Let me know if you think this causes any issues.

[aazuspan]

I decided to remove _raw.py and just rename _KNeighborsRegressor to RawKNNRegressor in _base.py. I think we were going to get caught in a circular dependency otherwise (RawKNNRegressor would rely on the mixins in _base, and then _TransformedKNeighborsRegressor would rely on _raw). Let me know if you think this causes any issues.

Yeah, I think you're right - this solution looks great to me. Nice to get rid of a module that wasn't adding much!

As far as I'm concerned, this looks ready to merge!

[grovduck]

@aazuspan, not that it's a huge deal, but I always manage to screw up the commit message when I merge things back in, so this one is a bit messy, but I didn't want to try to amend it. Hope it doesn't cause too much confusion.

[aazuspan]

No problem, it looks clear enough to me! And if anyone needs more details, they can always find the linked discussion here.