Early Stopping Tolerance

[aswer-svg]

Hi,
It's the first time I try this model and I want to understand what's going on. Here's how I've defined the model and fit:

from aplr import APLRRegressor

aplr_model = APLRRegressor(m=3000,
                           v=0.5,
                           loss_function="poisson",
                           link_function="log",
                           max_interaction_level=0,
                           random_state=42,
                           n_jobs=0,
                           faster_convergence=True,
                           verbosity=2
                           )
aplr_model.fit(X=X_train, y=y_train.to_numpy(), sample_weight=w_train.to_numpy())

Here's the result after 40 minutes (still training):

Fold: 0. Boosting step: 1. Model terms: 1. Terms eligible: 7. Validation error: 49.2768.
Fold: 0. Boosting step: 2. Model terms: 1. Terms eligible: 7. Validation error: 18.5956.
Fold: 0. Boosting step: 3. Model terms: 1. Terms eligible: 7. Validation error: 6.97092.
Fold: 0. Boosting step: 4. Model terms: 1. Terms eligible: 7. Validation error: 2.59861.
Fold: 0. Boosting step: 5. Model terms: 2. Terms eligible: 7. Validation error: 0.991619.
Fold: 0. Boosting step: 6. Model terms: 3. Terms eligible: 7. Validation error: 0.418974.
Fold: 0. Boosting step: 7. Model terms: 4. Terms eligible: 7. Validation error: 0.227364.
Fold: 0. Boosting step: 8. Model terms: 5. Terms eligible: 7. Validation error: 0.171765.
Fold: 0. Boosting step: 9. Model terms: 6. Terms eligible: 7. Validation error: 0.159438.
Fold: 0. Boosting step: 10. Model terms: 7. Terms eligible: 7. Validation error: 0.157471.
Fold: 0. Boosting step: 11. Model terms: 8. Terms eligible: 7. Validation error: 0.156974.
Fold: 0. Boosting step: 12. Model terms: 8. Terms eligible: 7. Validation error: 0.156715.
Fold: 0. Boosting step: 13. Model terms: 9. Terms eligible: 7. Validation error: 0.156494.
Fold: 0. Boosting step: 14. Model terms: 10. Terms eligible: 7. Validation error: 0.156375.
Fold: 0. Boosting step: 15. Model terms: 11. Terms eligible: 7. Validation error: 0.156665.
Fold: 0. Boosting step: 16. Model terms: 12. Terms eligible: 285. Validation error: 0.157648.
Fold: 0. Boosting step: 17. Model terms: 12. Terms eligible: 7. Validation error: 11.0938.
No further reduction in training loss was possible. Terminating the boosting procedure.
Model terms: 12. Terms available in final boosting step: 285.
Fold: 1. Boosting step: 1. Model terms: 1. Terms eligible: 7. Validation error: 49.2765.
Fold: 1. Boosting step: 2. Model terms: 1. Terms eligible: 7. Validation error: 18.5095.
Fold: 1. Boosting step: 3. Model terms: 1. Terms eligible: 7. Validation error: 6.90612.
Fold: 1. Boosting step: 4. Model terms: 1. Terms eligible: 7. Validation error: 2.56783.
Fold: 1. Boosting step: 5. Model terms: 2. Terms eligible: 7. Validation error: 0.979436.
Fold: 1. Boosting step: 6. Model terms: 3. Terms eligible: 7. Validation error: 0.414664.
Fold: 1. Boosting step: 7. Model terms: 4. Terms eligible: 7. Validation error: 0.226083.
Fold: 1. Boosting step: 8. Model terms: 5. Terms eligible: 7. Validation error: 0.171545.
Fold: 1. Boosting step: 9. Model terms: 6. Terms eligible: 7. Validation error: 0.159531.
Fold: 1. Boosting step: 10. Model terms: 7. Terms eligible: 7. Validation error: 0.157649.
Fold: 1. Boosting step: 11. Model terms: 8. Terms eligible: 7. Validation error: 0.157212.
Fold: 1. Boosting step: 12. Model terms: 8. Terms eligible: 7. Validation error: 0.156996.
Fold: 1. Boosting step: 13. Model terms: 9. Terms eligible: 7. Validation error: 0.156784.
Fold: 1. Boosting step: 14. Model terms: 10. Terms eligible: 7. Validation error: 0.156697.
Fold: 1. Boosting step: 15. Model terms: 11. Terms eligible: 7. Validation error: 0.156644.
Fold: 1. Boosting step: 16. Model terms: 12. Terms eligible: 285. Validation error: 0.156577.
Fold: 1. Boosting step: 17. Model terms: 12. Terms eligible: 7. Validation error: 11.1118.
No further reduction in training loss was possible. Terminating the boosting procedure.
Model terms: 12. Terms available in final boosting step: 285.
Fold: 2. Boosting step: 1. Model terms: 1. Terms eligible: 7. Validation error: 24.598.
Fold: 2. Boosting step: 2. Model terms: 1. Terms eligible: 7. Validation error: 9.09204.
Fold: 2. Boosting step: 3. Model terms: 1. Terms eligible: 7. Validation error: 3.36553.
Fold: 2. Boosting step: 4. Model terms: 2. Terms eligible: 7. Validation error: 1.26737.
Fold: 2. Boosting step: 5. Model terms: 3. Terms eligible: 7. Validation error: 0.515576.
Fold: 2. Boosting step: 6. Model terms: 4. Terms eligible: 7. Validation error: 0.259434.
Fold: 2. Boosting step: 7. Model terms: 5. Terms eligible: 7. Validation error: 0.181993.
Fold: 2. Boosting step: 8. Model terms: 6. Terms eligible: 7. Validation error: 0.163715.
Fold: 2. Boosting step: 9. Model terms: 7. Terms eligible: 7. Validation error: 0.160765.
Fold: 2. Boosting step: 10. Model terms: 8. Terms eligible: 7. Validation error: 0.160268.
Fold: 2. Boosting step: 11. Model terms: 9. Terms eligible: 7. Validation error: 0.160054.
Fold: 2. Boosting step: 12. Model terms: 10. Terms eligible: 7. Validation error: 0.159887.
Fold: 2. Boosting step: 13. Model terms: 11. Terms eligible: 7. Validation error: 0.15988.
Fold: 2. Boosting step: 14. Model terms: 12. Terms eligible: 7. Validation error: 0.15978.
Fold: 2. Boosting step: 15. Model terms: 12. Terms eligible: 7. Validation error: 11.4205.
No further reduction in training loss was possible. Terminating the boosting procedure.
Model terms: 12. Terms available in final boosting step: 285.
Fold: 3. Boosting step: 1. Model terms: 1. Terms eligible: 7. Validation error: 49.2754.
Fold: 3. Boosting step: 2. Model terms: 1. Terms eligible: 7. Validation error: 18.6532.
Fold: 3. Boosting step: 3. Model terms: 1. Terms eligible: 7. Validation error: 7.01459.
Fold: 3. Boosting step: 4. Model terms: 1. Terms eligible: 7. Validation error: 2.61954.
Fold: 3. Boosting step: 5. Model terms: 2. Terms eligible: 7. Validation error: 1.00013.
Fold: 3. Boosting step: 6. Model terms: 3. Terms eligible: 7. Validation error: 0.421837.
Fold: 3. Boosting step: 7. Model terms: 4. Terms eligible: 7. Validation error: 0.228616.
Fold: 3. Boosting step: 8. Model terms: 5. Terms eligible: 7. Validation error: 0.172648.
Fold: 3. Boosting step: 9. Model terms: 6. Terms eligible: 7. Validation error: 0.160335.
Fold: 3. Boosting step: 10. Model terms: 7. Terms eligible: 7. Validation error: 0.158438.
Fold: 3. Boosting step: 11. Model terms: 8. Terms eligible: 7. Validation error: 0.157929.
Fold: 3. Boosting step: 12. Model terms: 8. Terms eligible: 7. Validation error: 0.157667.
Fold: 3. Boosting step: 13. Model terms: 9. Terms eligible: 7. Validation error: 0.157488.
Fold: 3. Boosting step: 14. Model terms: 10. Terms eligible: 7. Validation error: 0.157343.
Fold: 3. Boosting step: 15. Model terms: 11. Terms eligible: 7. Validation error: 0.157409.
Fold: 3. Boosting step: 16. Model terms: 12. Terms eligible: 285. Validation error: 0.157272.
Fold: 3. Boosting step: 17. Model terms: 13. Terms eligible: 7. Validation error: 0.157272.
Fold: 3. Boosting step: 18. Model terms: 13. Terms eligible: 7. Validation error: 0.157272.
Fold: 3. Boosting step: 19. Model terms: 13. Terms eligible: 7. Validation error: 0.157271.
Fold: 3. Boosting step: 20. Model terms: 13. Terms eligible: 7. Validation error: 0.157271.
Fold: 3. Boosting step: 21. Model terms: 13. Terms eligible: 7. Validation error: 0.157271.
Fold: 3. Boosting step: 22. Model terms: 13. Terms eligible: 7. Validation error: 0.157271.
Fold: 3. Boosting step: 23. Model terms: 13. Terms eligible: 7. Validation error: 0.157271.
Fold: 3. Boosting step: 24. Model terms: 13. Terms eligible: 7. Validation error: 0.157271.
Fold: 3. Boosting step: 25. Model terms: 13. Terms eligible: 7. Validation error: 0.157271.
...
...
...
Fold: 3. Boosting step: 348. Model terms: 15. Terms eligible: 7. Validation error: 0.157243.
Fold: 3. Boosting step: 349. Model terms: 15. Terms eligible: 7. Validation error: 0.157242.
Fold: 3. Boosting step: 350. Model terms: 15. Terms eligible: 7. Validation error: 0.157242.
Fold: 3. Boosting step: 351. Model terms: 15. Terms eligible: 7. Validation error: 0.157242.
Fold: 3. Boosting step: 352. Model terms: 15. Terms eligible: 285. Validation error: 0.157242.
Fold: 3. Boosting step: 353. Model terms: 15. Terms eligible: 7. Validation error: 0.157242.
Fold: 3. Boosting step: 354. Model terms: 15. Terms eligible: 7. Validation error: 0.157242.
Fold: 3. Boosting step: 355. Model terms: 15. Terms eligible: 7. Validation error: 0.157242.
Fold: 3. Boosting step: 356. Model terms: 15. Terms eligible: 7. Validation error: 0.157242.

I have 2 questions about this:

-Is it possible to add early stopping tolerance? Why do the first 3 folds end so quickly, while the fourth one takes so much time? The whole model took about 5 hours to fit.

-Why do terms eligible shoot up every 15 rounds? I thought I had disabled interactions, why are there 285 terms eligible? I don't have that much columns.

[mathias-von-ottenbreit]

Hi.

In the API reference you can read about the following that should help you:

• early_stopping_rounds.
• Reducing the number of allowed boosting steps, m, can also be used to speed up the algorithm. So can max_terms.
• ineligible_boosting_steps_added and max_eligible_terms. This is why the number of eligible terms to be evaluated jumps every 15th boosting step. If you have string columns in X_train then they will automatically be one hot encoded. Each category is then treated as a potential predictor and included in the count of 285.

Regarding the early termination in the first folds I do not know your data and because of that it is difficult to give an accurate advice. The message "No further reduction in training loss was possible. Terminating the boosting procedure." means that the algorithm could not find a new term or update an existing term that further reduced the training loss.
Some things to potentially try:

• A lower learning rate, v, such as 0.1 and/or disable faster_convergence. This will likely increase training time but perhaps avoid the seemingly premature convergence in the first folds. Perhaps combine this with a lower m to reduce the fitting time.
• Use another loss_function instead of poisson, such as tweedie. Then dispersion_parameter should ideally be tuned (set validation_tuning_metric to something that is not affected by dispersion_parameter when tuning, such as mse). Without faster_convergence, tweedie with a high dispersion_parameter (such as 1.9) tends to converge significantly faster on some datasets.
• Your dataset is probably large. If you want to reduce training time then you can use a randomly selected subsample of it for training. Considering that you are not fitting interactions the reduction in dataset size may not necessarily degrade predictiveness.

Best regards,
Mathias