[python-package] custom early stopping by constraint

[sluo1250]

Hi there,

I'm trying to do early stopping based on a constraint. Specifically, I have a custom metric that I would like to keep under a threshold, but not trying to optimize. For example, I'm trying to do lgbm.train() with early stopping based on two metrics, one inherited from objective which is 'quantile', and the other one is my custom metric and I would like to keep quantile going less while keep my custom metric under 0.

I don't see any functionality/parameters to do described above, so I was trying to do a hack that if the calculated metric is less than 0, make it become 0, thus it satisifies "require the early stopping metric to improve by at least this delta to be considered an improvement", as described here early_stopping_min_delta. From my understand of early_stopping_min_delta, it should not stop when the score is the same across iterations when it is set to 0, but from what I'm observing it will stop (see below example when early_stopping_round=20), so I guess it has to improve.

Is there any way to directly do early stopping based on a threshold? If not what would be a suggested way to get around it?

Thanks in advance.

[1]     valid_0's quantile: 3.3088      valid_0's MIM: 0
[2]     valid_0's quantile: 3.29459     valid_0's MIM: 0
[3]     valid_0's quantile: 3.28052     valid_0's MIM: 0
[4]     valid_0's quantile: 3.26736     valid_0's MIM: 0
[5]     valid_0's quantile: 3.25439     valid_0's MIM: 0
[6]     valid_0's quantile: 3.24121     valid_0's MIM: 0
[7]     valid_0's quantile: 3.22818     valid_0's MIM: 0
[8]     valid_0's quantile: 3.21452     valid_0's MIM: 0
[9]     valid_0's quantile: 3.20201     valid_0's MIM: 0
[10]    valid_0's quantile: 3.19017     valid_0's MIM: 0
[11]    valid_0's quantile: 3.17816     valid_0's MIM: 0
[12]    valid_0's quantile: 3.16569     valid_0's MIM: 0
[13]    valid_0's quantile: 3.15328     valid_0's MIM: 0
[14]    valid_0's quantile: 3.14131     valid_0's MIM: 0
[15]    valid_0's quantile: 3.12919     valid_0's MIM: 0
[16]    valid_0's quantile: 3.1168      valid_0's MIM: 0
[17]    valid_0's quantile: 3.1045      valid_0's MIM: 0
[18]    valid_0's quantile: 3.09259     valid_0's MIM: 0
[19]    valid_0's quantile: 3.08005     valid_0's MIM: 0
[20]    valid_0's quantile: 3.06624     valid_0's MIM: 0
[21]    valid_0's quantile: 3.05351     valid_0's MIM: 0

[jameslamb]

Thanks for using LightGBM.

I think what you're asking is possible, but I want to be sure I understand before writing up an example.

Are you saying that you want to achieve the behavior "immediately stop training if {metric} > {threshold}"?

[sluo1250]

@jameslamb Yeah, that's exactly what I want.

[jameslamb]

Got it!

Ok yes, that is possible with only public parts of lightgbm's API, but definitely involves some things that are not well documented.

Here's an example:

import lightgbm as lgb
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

class EarlyStopOnThreshold:
    """
    Stop training immediately if a metric is falls below a certain value.
    """

    def __init__(self, metric_name: str, threshold: float):
        self.metric_name = metric_name
        self.threshold = threshold
        # ensure this is called after all other LightGBM callbacks
        self.order = 1000

    def __call__(self, env: lgb.callback.CallbackEnv):
        if env.evaluation_result_list:
            for dataset_name, metric_name, metric_value, *_ in env.evaluation_result_list:
                if metric_name == self.metric_name and metric_value < self.threshold:
                    log_msg = (
                        f"Stopping training because eval score '{metric_name} = {metric_value}' on dataset '{dataset_name}' "
                        f"is below the threshold ({self.threshold})."
                    )
                    print(log_msg)
                    raise lgb.EarlyStopException(
                        best_iteration=env.iteration,
                        best_score=env.evaluation_result_list
                    )

# create training and validation data
X, y = make_regression(n_samples=10_000, random_state=708, n_features=6, n_informative=6)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

dtrain = lgb.Dataset(X_train, label=y_train)
dvalid = dtrain.create_valid(data=X_valid, label=y_valid)

# train a model
bst = lgb.train(
    train_set=dtrain,
    params={
        "objective": "regression_l2",
        "metric": ["l1", "mape"],
        "num_iterations": 100,
        "seed": 708,
        "verbose": 1,
    },
    callbacks=[
        lgb.log_evaluation(),
        EarlyStopOnThreshold(metric_name="mape", threshold=0.80),
    ],
    valid_sets=[
        dvalid,
    ]
)

Run that with lightgbm>=4.6.0 and you should see output like the following:

[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000165 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 1530
[LightGBM] [Info] Number of data points in the train set: 7500, number of used features: 6
[LightGBM] [Info] Start training from score -1.021211
[1]     valid_0's l1: 55.5769   valid_0's mape: 0.96029
[2]     valid_0's l1: 51.5072   valid_0's mape: 0.93244
[3]     valid_0's l1: 47.8454   valid_0's mape: 0.913856
[4]     valid_0's l1: 44.6128   valid_0's mape: 0.896247
[5]     valid_0's l1: 41.5834   valid_0's mape: 0.884918
[6]     valid_0's l1: 38.8369   valid_0's mape: 0.869822
[7]     valid_0's l1: 36.3962   valid_0's mape: 0.853963
[8]     valid_0's l1: 34.1084   valid_0's mape: 0.837159
[9]     valid_0's l1: 32.066    valid_0's mape: 0.824543
[10]    valid_0's l1: 30.1848   valid_0's mape: 0.812867
[11]    valid_0's l1: 28.4463   valid_0's mape: 0.796366
Stopping training because eval score 'mape = 0.7963656908649135' on dataset 'valid_0' is below the threshold (0.8).

And should see that training really did stop after 11 boosting rounds (which is "early" because I used num_iterations=100).

bst.best_iteration
# 11

bst.best_score
# defaultdict(<class 'collections.OrderedDict'>, {'valid_0': OrderedDict([('l1', np.float64(28.063683801501885)), ('mape', np.float64(0.7814899988347561))])})

How this works and how to modify it

Raising an lgb.EarlyStoppingException from any callback triggers early stopping. That's handled by this check that runs on every iteration:

LightGBM/python-package/lightgbm/engine.py

Lines 342 to 345 in 72a3981

	except callback.EarlyStopException as earlyStopException:
	booster.best_iteration = earlyStopException.best_iteration + 1
	evaluation_result_list = earlyStopException.best_score
	break

Callbacks accept a single value, env, that tracks state about the training process. You can see the available fields and their types here:

LightGBM/python-package/lightgbm/callback.py

Lines 64 to 71 in 72a3981

	@dataclass
	class CallbackEnv:
	model: Union[Booster, "CVBooster"]
	params: Dict[str, Any]
	iteration: int
	begin_iteration: int
	end_iteration: int
	evaluation_result_list: Optional[_ListOfEvalResultTuples]

Hopefully this is enough information for you to put together a solution with the behavior you want.

[sluo1250]

@jameslamb wow this is clean! Thanks for your detailed example. It looks like you get rid of lightgbm.early_stopping and instead use a custom callback. So if I want to have, say "l1" to function normally, and "mape" to stop based on a threshold, the best practice would be to copy _EarlyStoppingCallback here and do something like

if metric_name == 'mape':
    if metric_value < 0.8:
                   raise lgb.EarlyStopException(
                        best_iteration=env.iteration,
                        best_score=env.evaluation_result_list
                    )

correct?

[jameslamb]

So if I want to have, say "l1" to function normally

What specifically do you mean by "function normally"? Are you saying you want something like this?

• early stop if l1 does not improve for {early_stopping_rounds} consecutive rounds
• early stop immediately if mape < 0.8

[sluo1250]

@jameslamb that's correct