Does one need to reset Metrics during the end of epoch in Lightning 1.0.3

[junwen-austin]

❓ Questions and Help

Before asking:

What is your question?

Given the new metrics in 1.0.0 and later (which I really like!), I have three accuracy metrics for training, validation and test initialized in __init__ function. Do I need to reset them at the end of training and validation epoch given they will be used multiple times?

Code

What have you tried?

What's your environment?

• OS: [e.g. iOS, Linux, Win]
• Packaging [e.g. pip, conda]
• Version [e.g. 0.5.2.1]

[SkafteNicki]

Depends on how you are using the metrics. In general if the .compute()method is called the internal state is reset. This means that if you call .compute() in the end off the epoch you should be fine. If you are using metrics in combination with self.log, then setting on_epoch=True will also internally call . compute() at the end of the epoch.

[junwen-austin]

@SkafteNicki Thanks for the information. This has been very useful.

[junwen-austin]

I'd like to get the validation accuracy over the entire validation data and I have seen some strange results on it with DDP. I want to make sure I am doing the right thing with metrics with DDP. Below is my psedu-code:

def __init__(self):
    self.val_acc = Accuracy(compute_on_step=False)

def validation_step(self, batch, batch_idx):
      input, y = batch[0], batch[1]
      logits = self(input)
      _, pred = torch.max(logits, dim=1)
     self.val_acc.update(pred, y)
     self.log('val_acc', self.val_acc, on_step=False, on_epoch=True)

Could someone tell if this implementation is to give me validation accuracy for the entire validation data? Thanks.

[SkafteNicki]

Yes that is the correct way.
Could you explain the strange results you are seeing in ddp mode?

[LittlePea13]

I am facing similar issues with ddp. When computing F1 using Fbeta, if I use ddp, the results are not the same as running them in dp (in a single machine). To compare I use the same saved checkpoint. In my case the metric is computed in test_step_end:
def test_step_end(self, test_step_outputs):

def test_step_end(self, test_step_outputs):
    self.fbeta_test(test_step_outputs['y_hat'], test_step_outputs['y'])

without logging. And then in test_epoch_end:

def test_epoch_end(self, outputs: list):
    fbeta_test = self.fbeta_test.compute()

So I am not sure if in ddp is just averaging the metric across ranks instead of computing the metric over the whole dataset. It is worth saying that when computing the metric on test_epoch_end, the value fbeta_test is the same across ranks. With dp the result is .5670 and with ddp it returns .6154. Running it on a single gpu with no dp returns the same value as dp (.5670), which I assume is the correct one.

[junwen-austin]

@SkafteNicki I am trying a new model with PyTorch Lightning and also with the new metrics in 1.0.3. The strange metric results in DDP may or may not be related to the new metrics. That's why I'd like to make sure I am doing the right thing with the new metrics. This helps me debug any issue that might be related to the new model. Thanks.

[SkafteNicki]

Thanks @LittlePea13 and @junwen-austin for both getting back to me.
It seems to me that there may be a problem with how metrics are aggregated in ddp mode. I will try to identify the issue and get back to you.

[LittlePea13]

No problem @SkafteNicki. Let me know if I can help in any way, I tried to debug a bit but didn't get very far, I am quite new to ddp. And thanks for your work :)

[SkafteNicki]

@LittlePea13 could you try setting the dist_sync_on_step to Trueand see if it solved the problem?
If so then it have something to do with how buffer metric states.

[LittlePea13]

@SkafteNicki I just tried and the result was still the same (wrong) by declaring the metric as:
self.fbeta_test = Fbeta(num_classes=1, dist_sync_on_step = True)

[hoanghng]

Depends on how you are using the metrics. In general if the .compute()method is called the internal state is reset. This means that if you call .compute() in the end off the epoch you should be fine. If you are using metrics in combination with self.log, then setting on_epoch=True will also internally call . compute() at the end of the epoch.

Overwriting on_epoch_end allows computing metrics using all data, but there is no way to log those results (or even use it for model selection. It's because log_train_epoch_end_metrics is called right after the loop through batches and before on_epoch_end.
https://github.com/PyTorchLightning/pytorch-lightning/blob/b50dd12332bf83209d9535c8516486edc1a6b252/pytorch_lightning/trainer/training_loop.py#L608-L613

[SkafteNicki]

@hoanghng instead of using on_epoch_end could you use training_epoch_end? That should work with logging.

[hoanghng]

@SkafteNicki Thanks a lot. It works like a charm.