Early stopping for XGBoost + Update Readme

[inventormc]

This PR supports early stopping for XGBoost. We leverage the incremental learning capabilities of XGBoost:

Note that this may not necessarily improve performance but instead allows us to break the training process into multiple parts.

clf = XGBClassifier(n_estimators=10, nthread=8)
base_model = None
for i in range(20):
    z = clf.fit(x_tr, y_tr, xgb_model=base_model)
    y_te = z.predict(x_te)
    print(sklearn.metrics.mean_squared_error(y_te, y_pr))
    base_model = z.get_booster()

resolves #58

[inventormc]

microsoft/LightGBM#3057 -init_model doesn't exist in the latest stable release yet

[richardliaw]

what if you just put this in another trainable? would it make sense there?

[inventormc]

Why do we need to put it into another trainable?

[richardliaw]

i guess it's probably going to be unsustainable to have like 5 different special cases for different libraries, but right now it's probably fine

[richardliaw]

OK this looks good to me. I've added a couple updates to make things clearer.

[inventormc]

is this supposed to be under the case the we can partial_fit? I think xgboost doesn't have the that method, so maybe this should be outside

[richardliaw]

OK nice, I wrote a couple tests here.

[inventormc]

I think there's actually more cases we need to be checking here. Here's all the cases:

• User wants to early stop, and it can be done
  • need to make sure the user sets max_iters, so raise a warning if this isn't done
• User wants to early stop, and it cannot be done
  • directly throw an error, regardless of what max_iters is
  • maybe it would be nice the remove the warning that would come up in the if not self._can_early_stop() and max_iters > 1: since there will be duplicate errors
• User does not want to early stop, and it can be done
  • if max_iters isn't set, do nothing
  • if max_iters is set, raise a warning that it is ignored because user didn't enable early stop
• User does not want to early stop, and it cannot be done
  • if max_iters isn't set, do nothing
  • if max_iters is set, raise a warning that it is ignored because user didn't enable early stop

Right now

[richardliaw]

It is now reduced to this:

• User wants to early stop, and it cannot be done
  • directly throw an error.
• User wants to early stop, and it can be done, and max_iters is not set
  • raise a warning that it should be set.
• regardless of whether it can be done, if user does not want to early stop and if max_iters is set:
  • raise a warning that it is ignored / set to 1.