ResNeXT OOM Error

[ysharma1126]

Get OOM error evaluating ResNeXT model even with batch size = 10

[ppwwyyxx]

Then perhaps you need to use a smaller batch size.

[ysharma1126]

I am using a GTX 1080 Ti, and was told your CAAD 2018 submission used a batch size of 20. Given that, I am wondering why the batch size needs to be reduced that significantly.

[ppwwyyxx]

It's best to paste your command so we know what you did.
Are you using the --eval-directory flag for main.py? If that's the case you actually need to modify the code to change its batch size.

[ysharma1126]

I am performing white-box evaluation on a single GPU. Note that I do not have trouble with the default batch size of 32 when evaluating R152. For ResNeXT, manually reducing the batch size to 10 still yields OOM errors.

python main.py --eval --load X101-DenoiseAll.npz --data [ImageNet_Dir] --attack-iter 1 --attack-epsilon 16.0 --arch ResNeXtDenoiseAll -d 101

[ppwwyyxx]

In the CAAD submission, only forward pass is needed so the memory requirement there is very minimal.

On a GPU with 16G memory I can run your command with 32 batch size while seeing some OOM messages (which did not affect the evaluation).

[ysharma1126]

Understood, my evaluation is terminated due to ResourceExhaustedError. I will reduce the batch size further. Thanks.

[ppwwyyxx]

I grabbed a 1080Ti with 10.4G free memory and I was able to execute your command with batch size 15.

It's probably just an environment issue.

I'm on TF1.13.0rc1, cuda10, cudnn7.4.24, if that matters.