Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[python-package] Early Stopping does not work in LGBMClassifier #6244

Closed
ohbtorres opened this issue Dec 21, 2023 · 6 comments · Fixed by #6846
Closed

[python-package] Early Stopping does not work in LGBMClassifier #6244

ohbtorres opened this issue Dec 21, 2023 · 6 comments · Fixed by #6846
Assignees
@jameslamb
Labels
question

Comments

@ohbtorres
Copy link

Description

Using lightgbm.early_stopping does not work while using Scikit-learn API, and the training still remains after the early stop condition occurs

Reproducible example

import pandas as pd
import numpy as np
import lightgbm as lgbm

# Random train set
x12 = pd.DataFrame({"x1": np.random.randint(-10, 10, 10000),
                                   "x2": np.random.randint(-10, 10, 10000)})
y12 = pd.Series(np.random.randint(0, 2, 10000))
# Random validation set
x12_valid = pd.DataFrame({"x1": np.random.randint(-10, 10, 1000),
                          "x2": np.random.randint(-10, 10, 1000)})
y12_valid = pd.Series(np.random.randint(0, 2, 1000))
# Training the model using a validation set
lgbm.LGBMClassifier(boosting_type='dart',
                    learning_rate=0.001,
                    n_estimators=5000,
                    objective='binary')\
    .fit(x12, y12,
         eval_set=(x12_valid, y12_valid),
         eval_metric="logloss",
         callbacks=[lgbm.log_evaluation(period=100),
                          lgbm.early_stopping(stopping_rounds=100)])

The result will be this
image

The binary_logloss starts with 0.693169 and end with 0.697759 and no stopping happened

Environment info

LightGBM version or commit hash: 4.1.0

Command(s) you used to install LightGBM

pip install lightgbm

pandas==2.0.3
numpy==1.24.4
@jmoralez
Copy link
Collaborator

Hey @ohbtorres, thanks for using LightGBM. Early stopping isn't supported in dart, you should be getting this warning:

LightGBM/python-package/lightgbm/callback.py

Line 325 in 0a9a6bb

_log_warning('Early stopping is not available in dart mode')

Is it not showing?

@ohbtorres
Copy link
Author

ohbtorres commented Dec 21, 2023 •
edited
Loading

Hi @jmoralez, thank you for your answer.
You are right, changing the boosting_type the callback works. Thanks a lot!
Is there any information about this here?
I am not getting this warning message
image

@jmoralez
Copy link
Collaborator

Is there any information about this here?

I think we could add it here.

I am not getting this warning message

Did you disable Python's warnings? Running your example I get the following:

[LightGBM] [Info] Number of positive: 5074, number of negative: 4926
[LightGBM] [Info] Auto-choosing row-wise multi-threading, the overhead of testing was 0.000097 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 40
[LightGBM] [Info] Number of data points in the train set: 10000, number of used features: 2
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.507400 -> initscore=0.029602
[LightGBM] [Info] Start training from score 0.029602
/hdd/github/LightGBM/python-package/lightgbm/callback.py:325: UserWarning: Early stopping is not available in dart mode
  _log_warning('Early stopping is not available in dart mode')

@ohbtorres
Copy link
Author

I didn't. I don't know what coulb be happening.
However, I got it, and early stopping is working fine. Thanks @jmoralez!

@jmoralez
Copy link
Collaborator

jmoralez commented Dec 21, 2023 •
edited
Loading

Would you like to add that dart isn't supported in the early stopping callback docstring?

@jameslamb jameslamb changed the title Early Stoppinf does not work in LGBMClassifier Early Stopping does not work in LGBMClassifier Dec 22, 2023
@jameslamb jameslamb changed the title Early Stopping does not work in LGBMClassifier [python-package] Early Stopping does not work in LGBMClassifier Feb 28, 2025
@jameslamb jameslamb mentioned this issue Feb 28, 2025
Merged
@jameslamb
Copy link
Collaborator

Would you like to add that dart isn't supported in the early stopping callback docstring?

It's been over a year without a response here, so I put up a PR adding that note: #6846

@jameslamb jameslamb self-assigned this Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jameslamb jameslamb

Labels
question
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[docs] add a note about DART in early_stopping docstring microsoft/LightGBM
3 participants
@jameslamb @jmoralez @ohbtorres