Increased memory usage vs. torchvision equivalent

[nlml]

Hello,

First of all: fantastic paper and contribution -- and the pypi package is the cherry on top :D

I decided to try switching one of my model trainings to use antialiased_cnns.resnet34 as a drop-in replacement for torchvision.models.resnet34. It seems however that the memory needs are almost 1.5x higher with the anti-aliased CNN. This is based on the fact that with the torchvision version, my model trains with a batch size of 16 per GPU (it's a sequence model, so the actual number of images going through the CNN per batch is actually much higher). With the anti-aliased CNN, I get CUDA out of memory errors for any batch size above 11.

Were you aware of this? I'm not really expecting you to post a fix, just wondering if it makes sense to you and if you were already aware of it.

Thanks again!

[richzhang]

Thanks for the message! Yes, it takes more memory. I implemented a gradient accumulation flag, which approximates training with larger batch sizes (for example, accumulate the gradient over 2 batches of size N/2 and then update)

[nlml]

Thanks for the response! Out of curiosity, did you think about whether the memory footprint can be reduced, maybe by using some in-place operations, or is that not possible here?

[richzhang]

I think if it were forward-pass only, the operation could be made in-place