[Feature request] Early stopping doesn't work with XGBoost, LightGBM or CatBoost

[rohan-gt]

I'm getting the following error while setting early_stopping=True

ValueError: Early stopping is not supported because the estimator does not have `partial_fit`

These could be a potential fixes:

1. XGBoost: incremental learning, partial_fit like sklearn? dmlc/xgboost#1686
2. LightGBM: Is there a way to calling fit() multiple times on the same model and stay the previous fitted stuff like the partial_fit() in some sklearn classifiers. microsoft/LightGBM#2718
3. CatBoost: Is incremental training supported? catboost/catboost#464

[richardliaw]

Yeah, that's a good point. I think we'll probably want to special-case a training call for xgboost, lightgbm as they have a different way of doing early stopping..

[rohan-gt]

@richardliaw I've updated the issue with more details. Btw how does max_iters perform early stopping? I'm a little confused about how it works in conjunction with n_iter

[richardliaw]

yeah... i guess that's the penalty we have to pay for adhering to the sklearn API.

max_iters = number of "epochs"
n_iter = number of hyperparameter evals.

Does that make sense?

[richardliaw]

In #63, we're going to enable early stopping for XGBoost via incremental learning. We decided not to implement it for lgbm because it is not yet on a stable version.

[richardliaw]

Hmm, not sure how we're going to support CatBoost but will open an issue to track lightgbm.

[rohan-gt]

@richardliaw is early stopping enabled using cross-validation like I mentioned here? Because CatBoost has a cv() method too