When is `on_validation_epoch_start` / `on_validation_epoch_end` being called?

[siahuat0727]

❓ Questions and Help

What is your question?

When is on_validation_epoch_start / on_validation_epoch_end being called?
It there a doc that explains the order of the callback function being called?

I need a callback function that will be called at the end of every validation epoch.
I read the callback docs and I think on_validation_epoch_end is the target function I need to override. But after my trial and error, I found that it is on_validation_end that meets my needs.

What have you tried?

And I found that on the latest version, nobody will call on_validation_epoch_start except the function on_validation_epoch_start itself in "callback_hook.py". Is it means that on_validation_epoch_start will never be called?

~/pytorch-lightning$ grep -rH "on_validation_epoch_start"
pytorch_lightning/core/hooks.py:    def on_validation_epoch_start(self) -> None:
pytorch_lightning/callbacks/base.py:    def on_validation_epoch_start(self, trainer, pl_module):
pytorch_lightning/trainer/callback_hook.py:    def on_validation_epoch_start(self):
pytorch_lightning/trainer/callback_hook.py:            callback.on_validation_epoch_start(self, self.get_model())

What's your environment?

• OS: Linux
• Packaging: pip
• Version: 0.9.0

[github-actions]

Hi! thanks for your contribution!, great first issue!

[rohitgr7]

It's been called:
https://github.com/PyTorchLightning/pytorch-lightning/blob/ad0f1194aa2fa8cc82c915e49aca3a1149901709/pytorch_lightning/trainer/evaluation_loop.py#L287-L290
https://github.com/PyTorchLightning/pytorch-lightning/blob/ad0f1194aa2fa8cc82c915e49aca3a1149901709/pytorch_lightning/trainer/evaluation_loop.py#L232-L252

[siahuat0727]

@rohitgr7

Hihi, thanks a lot!

Before digging into the code, I tried to override a Callback class as follow and fit Trainer with 3 epochs, then I got the output as follow:

import inspect

class Bar(Callback):

    def setup(self, trainer, stage: str):
        print(f'\n\n{inspect.currentframe().f_code.co_name}\n\n')

    def teardown(self, trainer, stage: str):
        print(f'\n\n{inspect.currentframe().f_code.co_name}\n\n')

    def on_init_start(self, trainer):
        print(f'\n\n{inspect.currentframe().f_code.co_name}\n\n')

    def on_init_end(self, trainer):
        print(f'\n\n{inspect.currentframe().f_code.co_name}\n\n')

    ...

on_init_start
on_init_end

on_fit_start
setup
  on_sanity_check_start
  on_sanity_check_end
  on_train_start

    on_epoch_start
    on_validation_start
    on_validation_end
    on_epoch_end

    on_epoch_start
    on_validation_start
    on_validation_end
    on_epoch_end

    on_epoch_start
    on_validation_start
    on_validation_end
    on_epoch_end

  on_train_end
on_fit_end
teardown

setup
on_fit_start
setup
on_test_start
on_test_end
on_fit_end
teardown
teardown

I think on_train_start shoulder be paired with on_validation_start, and on_epoch_start should be paired with on_validation_epoch_start?

on_validation_epoch_start and on_validation_epoch_end are missing here.
Is it a bug?

[rohitgr7]

on_validation_epoch_start should be called after on_validation_start. Can you put all this in a colab notebook? I will check there :)

[siahuat0727]

Code

import os
import inspect

import torch
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST
from torchvision import transforms
import pytorch_lightning as pl
from pytorch_lightning import Trainer
from pytorch_lightning.callbacks.base import Callback


class LitModel(pl.LightningModule):

    def __init__(self, classes=10):
        super().__init__()
        self.l1 = torch.nn.Linear(28 * 28, classes)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = F.cross_entropy(y_hat, y)
        tensorboard_logs = {'train_loss': loss}
        return {'loss': loss, 'log': tensorboard_logs}

    def validation_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = F.cross_entropy(y_hat, y)
        return {'val_loss': loss}

    def validation_epoch_end(self, outputs):
        avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        return {'val_loss': avg_loss}

    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.001)


class Bar(Callback):
    def on_train_epoch_start(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_train_epoch_end(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_validation_epoch_start(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_validation_epoch_end(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_epoch_start(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_epoch_end(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_train_start(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_train_end(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_validation_start(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')

    def on_validation_end(self, trainer, pl_module):
        print(f'{inspect.currentframe().f_code.co_name}')


mnist_train = MNIST(os.getcwd(), train=True, download=True,
                    transform=transforms.ToTensor())
mnist_train = DataLoader(mnist_train, num_workers=1)
mnist_val = MNIST(os.getcwd(), train=False, download=False,
                  transform=transforms.ToTensor())
mnist_val = DataLoader(mnist_val, num_workers=1)

model = LitModel()

trainer = Trainer(callbacks=[Bar()], max_epochs=2, limit_train_batches=2,
                  limit_val_batches=2, progress_bar_refresh_rate=0)
trainer.fit(model, mnist_train, mnist_val)

Output

on_train_start
on_epoch_start
on_validation_start
on_validation_end
on_epoch_end
on_epoch_start
on_validation_start
on_validation_end
on_epoch_end
on_train_end

With this example, no on_train_epoch_start, on_train_epoch_end, on_validation_epoch_start and on_validation_epoch_end.

[rohitgr7]

Try >0.9.0rc2 or master. It's fixed there.

on_validation_epoch_start
on_validation_epoch_end
on_train_start
on_epoch_start
on_train_epoch_start
on_validation_start
on_validation_epoch_start
on_validation_epoch_end
on_validation_end
on_epoch_end
on_train_epoch_end
on_epoch_start
on_train_epoch_start
on_validation_start
on_validation_epoch_start
on_validation_epoch_end
on_validation_end
on_epoch_end
on_train_epoch_end
on_train_end

[siahuat0727]

@rohitgr7

I see, thanks!!

[prateeky2806]

@rohitgr7, seems like the second last (on_train_epoch_end) and third last (on_epoch_end) lines are swapped. Can you please check?

Also is there some place where the flow for all of these hooks is given in the lightening documentation?

Try >0.9.0rc2 or master. It's fixed there.

on_validation_epoch_start
on_validation_epoch_end
on_train_start
on_epoch_start
on_train_epoch_start
on_validation_start
on_validation_epoch_start
on_validation_epoch_end
on_validation_end
on_epoch_end
on_train_epoch_end
on_epoch_start
on_train_epoch_start
on_validation_start
on_validation_epoch_start
on_validation_epoch_end
on_validation_end
on_epoch_end
on_train_epoch_end
on_train_end

[akihironitta]

Hi @prateeky2806, it's documented at https://pytorch-lightning.readthedocs.io/en/1.5.10/common/lightning_module.html#hooks.