Make sure meta-estimators are lenient towards missing data

[adrinjalali]

This item is in our roadmap, and I don't totally understand it. Trying to kinda track the progress of those items, I'm creating this issue. Not sure who wrote it. @amueller maybe you could elaborate? (feel free to edit the description of the issue).

[glemaitre]

I think that we developed some meta-estimators which would call check_X_y and not allowing missing-values. However, this check can be done by the underlying estimator instead.

Let's give an example: If BaggingClassifier with an HistGradientBoostingClassifier as a base estimator. The gradient boosting will natively manage missing values. If the BaggingClassifier already raises an error at fit, it would be wrong.

So the idea is to delay this checking.

[jnothman]

This is also part of #9854

[jnothman]

Also related: #12072

[jnothman]

I think looking through our implementations to check which meta-estimators and ensembles enforce finiteness is pretty straightforward, so I've tagged this as such.

[NicolasHug]

IMHO, this is not an easy task from the perspective of new or inexperienced contributors (who are attracted by the help wanted tag).

Seeing something tagged as easy when it doesn't feel like it can be quite discouraging, and since our barrier of entry is already crazy high, I think we need to be careful when tagging issues as such

[jnothman]

I usually consider "easy" to require a grade of familiarity higher than "good first issue"

[venkyyuvy]

Im interested in working for this issue.

Some pointers would be helpful

[glemaitre]

@venkyyuvy The idea is "passthrough" NaN values. Since there are scikit-learn compatible estimator which internally manage missing values, it could be good that our preprocessor does not raise any error but instead discard and propagate missing values.

[yashika51]

Hi, I am interested in working on this issue.
Which estimators should be able to pass NaN?

[jnothman]

Basic idea is to look at ensemble, multioutput, etc and make sure that they do not validate their input beyond what is required for their algorithm to work

[yashika51]

So just to confirm, the estimators should be able to pass NaNs and not raise any error at check_X_y. And the base estimators should be able to handle these missing values(and they already do that).
Any changes will be made to estimators to pass these values.

[venkyyuvy]

Looks like bagging regressor is already lenient towards missing values.

Reference

I tried the following test and it is passing:

def test_leniency_for_missing_data():
    rng = np.random.RandomState(42)
    X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target,
                                                        random_state=rng)

    mask = np.random.choice([1, 0], X_train.shape, p=[.1, .9]).astype(bool)
    X_train[mask] = np.nan
    hgbc = HistGradientBoostingClassifier(max_iter=3)
    clf = BaggingClassifier(base_estimator=hgbc)
    clf.fit(X_train, y_train)

    assert clf.score(X_test, y_test) > 0.8

Probably I should start with other meta estimators?

Update:

Even VotingClassifier and StakingClassifier is working fine with missing values in X.

def test_leniency_for_missing_data():
    rng = np.random.RandomState(42)
    X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                        random_state=rng)

    mask = np.random.choice([1, 0], X_train.shape, p=[.1, .9]).astype(bool)
    X_train[mask] = np.nan
    hgbc = HistGradientBoostingClassifier(max_iter=3)
    clf = VotingClassifier(estimators=[('hgbc_1', hgbc), ('hgbc_2', hgbc)])
    clf.fit(X_train, y_train)

    assert clf.score(X_test, y_test) > 0.8

    clf = StackingClassifier(estimators=[('hgbc_1', hgbc), ('hgbc_2', hgbc)])
    clf.fit(X_train, y_train)
    assert clf.score(X_test, y_test) > 0.8

[jnothman]

If there is no existing test for it, please add it.

[adrinjalali]

@glemaitre does the PR indeed close this issue? I couldn't figure from the conversation there whether it fixes all the meta estimators or not.

[glemaitre]

I would say yes if we did not miss any: #17987 (comment)

[glemaitre]

Now we can move forward to run the common tests which is a bit more tricky

…

On Wed, 26 Aug 2020 at 20:44, Adrin Jalali ***@***.***> wrote: @glemaitre <https://github.com/glemaitre> does the PR indeed close this issue? I couldn't figure from the conversation there whether it fixes all the meta estimators or not. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15319 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABY32PZHHWN4NYG2MQER2R3SCVJYXANCNFSM4JDBNZJA> .

-- Guillaume Lemaitre Scikit-learn @ Inria Foundation https://glemaitre.github.io/