Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Langevin Parameter in CatBoost Tuning #142

Open
EmotionEngineer opened this issue Nov 10, 2023 · 2 comments
Open

Support for Langevin Parameter in CatBoost Tuning #142

EmotionEngineer opened this issue Nov 10, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@EmotionEngineer
Copy link

🚀 Feature Request

Motivation

I would like to introduce support for the langevin=True parameter in LightAutoML. This parameter is the Stochastic Gradient Langevin Boosting (SGLB) method, which is a powerful and efficient machine learning framework capable of handling a wide range of loss functions and providing provable generalization guarantees. The method is based on a special form of the Langevin diffusion equation specifically designed for gradient boosting. This allows us to theoretically guarantee global convergence even for multimodal loss functions, while standard gradient boosting algorithms can only guarantee local optimum Paper

Proposal

I propose that LightAutoML support the langevin=True parameter during hyperparameter tuning of CatBoost models. This would allow users to leverage the benefits of SGLB when tuning CatBoost models using LightAutoML.

Alternatives

As an alternative, users could manually set the langevin parameter when creating a CatBoost instance. However, this could be less convenient and efficient than having LightAutoML automatically tune the parameters.

Additional context

I have successfully used the langevin=True parameter during a Kaggle competition. This experience has shown me the potential benefits of this parameter, and I believe it would be beneficial to have this feature in LightAutoML.

More details: #5 Solution

@EmotionEngineer EmotionEngineer added the enhancement New feature or request label Nov 10, 2023
@alexmryzhkov
Copy link
Collaborator

@EmotionEngineer LightAutoML solution has almost the same score as yours, proof link, but we'll take a look.

@EmotionEngineer
Copy link
Author

@alexmryzhkov Hi!
I agree with you, but it is still possible to get a marginal gain and it may be worth a try. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants