`auto_lr_find` does not properly set configure the `Trainer` to use the suggested learning rate: intended or not?

[sorenmacbeth]

When using auto_lr_find: True in the TrainerConfig, the discovered 'suggested' learning rate is not set or updated in the Trainer.

BaseModel.train calls self._prepare_for_training() at L649 which in turn calls self._prepare_trainer where the Trainer object is configured. This sets the learning_rate for all optimizers to TrainerConfig.learning_rate before the auto_lr_find block at L655.

The PL Tuner's lr_find method by default updates the learning rate hparam, but since the Trainer and the optimizer have already been configured, this change to hparams doesn't get reflected during training.

So: Is this intended behavior or a bug? If it not intended, it seems to me that self._prepare_for_training() should be called again after auto_lr_find completes, somewhere around L676.

I am going to assume its a bug and submit a PR as described above. If the intention is NOT to update the learning rate using the suggested LR then I propose the documentation be updated to clarify this and update_attrs: False parameter be passed to Tuner.lr_find to supress the log line saying that the learning rate has been updated.

edit:
I attempted a PR, but just calling _prepare_for_training() or _prepare_trainer() is not enough since the optimizers are configured in TabularModel.prepare_model() and its params are not available to the train method as currently written. I'm not sure of the best approach to fixing this if it is a bug but I would be happy to submit a PR if @manujosephv or someone has ideas about how best to update the optimizers to use the suggested LR.

Cheers

[dosubot]

Hey there, @sorenmacbeth! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together!

The behavior you're describing, where the auto_lr_find does not automatically update the learning rate in the Trainer after finding the suggested learning rate, seems to be a gap between the expected and actual functionality. According to the documentation, the auto_lr_find parameter is intended to find an optimal learning rate, but it does not explicitly state that it will automatically update the Trainer with this learning rate ^[1][2].

If the intention is for the learning rate to be updated automatically, then your suggestion to call self._prepare_for_training() again after auto_lr_find completes seems reasonable. This would ensure that the Trainer and optimizers are configured with the newly suggested learning rate.

If the behavior is intended not to update the learning rate automatically, then updating the documentation to clarify this would be beneficial. Additionally, using the update_attrs: False parameter in Tuner.lr_find to prevent misleading log messages would also be a good approach.

Your proposal to submit a PR to address this issue aligns with these considerations.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.