You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After the validation step, the model is not set back to train mode because the following hook
def on_validation_model_train(self) -> None:
"""Called when the validation loop ends.
The validation loop by default restores the `training` mode of the LightningModule to what it was before
starting validation. Override this hook to change the behavior. See also
:meth:`~pytorch_lightning.core.hooks.ModelHooks.on_validation_model_eval`.
"""
# The loop won't call this hook unless it is overridden. The line below is here in case the user calls super().
self.trainer.model.train()
is not called as can be seen here:
def _on_evaluation_model_train(self) -> None:
"""Undoes the eval mode."""
trainer = self.trainer
hook_name = "on_test_model_train" if trainer.testing else "on_validation_model_train"
if is_overridden(hook_name, trainer.lightning_module):
call._call_lightning_module_hook(trainer, hook_name)
else:
self._module_mode.restore(trainer.lightning_module)
I don't see the point of this behavior and I think it is very likely to cause bugs if people start implementing their own mode logic and expect it to be called. Would be happy to understand this (:
What version are you seeing the problem on?
v2.4
How to reproduce the bug
def train(self, mode=True):
super().train(mode)
if mode:
print("set to training mode")
else:
print("set to evaluation mode")
Error messages and logs
Epoch 0: 100%
set to evaluation mode
Epoch 1: 27%
Environment
Current environment
#- PyTorch Lightning Version (e.g., 2.4.0):
#- PyTorch Version (e.g., 2.4):
#- Python version (e.g., 3.12):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
More info
No response
The text was updated successfully, but these errors were encountered:
ClemensSchwarke
changed the title
Trainer does not switch to train mode after validation step.
Trainer does not switch to train mode after validation step
Aug 7, 2024
Thanks for the explanation! I guess it makes sense to not override the mode of children. However, how would one then implement my print example for a module? I.e. how could I make sure the state of a module is actually changed as opposed to just the boolean flag? I want to use a different image augmentation in my module depending on the mode.
Bug description
After the validation step, the model is not set back to train mode because the following hook
is not called as can be seen here:
I don't see the point of this behavior and I think it is very likely to cause bugs if people start implementing their own mode logic and expect it to be called. Would be happy to understand this (:
What version are you seeing the problem on?
v2.4
How to reproduce the bug
Error messages and logs
Environment
Current environment
More info
No response
The text was updated successfully, but these errors were encountered: