Backup running statistics is incorrect

[denizetkar]

When MetaBatchNormLayer is called forward with backup_running_statistics=True, the running statistics are meant to be copied into the backup variables by copying:

HowToTrainYourMAMLPytorch/meta_neural_network_architectures.py

Line 241 in 8696470

self.backup_running_mean.data = copy(self.running_mean.data)

However, this is not what happens and the underlying data ends up being tied with each other such that when the running statistics get updated, so does the backup. Here is a short code snippet that minimally reproduces the same behavior:

import torch as th
from copy import copy

t = th.tensor([1, 2, 3])
t2 = th.empty_like(t)
t2.data = copy(t.data)
t[0] = 5
assert t2[0] == 5

Is my understanding of backing up the running statistics wrong or is this a bug that needs fixing?

[AntreasAntoniou]

I'll need some time to look into this, but I do believe that the behaviour I currently have coded is intentional.

I want the running statistics to be updated within a given episode, but then scrapped at the end if this isn't a training iteration.

[DubiousCactus]

@denizetkar is right, that's not your intended behaviour and the backup is overwritten by the validation pass. You simply need to use deepcopy instead.