Out of GPU memory

[lzgahbb]

Hello Hao,
Could tell me how many GPU memories will be used when you trained this model? I use Titan xp will cause out of memory when batch size larger than 4, but when I set batch size 4, it would meet another error when executes hard sample mining, the reason is all the 4 images from the sample identity. By the way, I trained with market1501.

[lzgahbb]

Hello Hao,
Could tell me how many GPU memories will be used when you trained this model? I use Titan xp will cause out of memory when batch size larger than 4, but when I set batch size 4, it would meet another error when executes hard sample mining, the reason is all the 4 images from the sample identity. By the way, I trained with market1501.

It still out of memory when I roll back to pytorch 0.4.0

[michuanhaohao]

@lzgahbb I think there exist some problems in your machine. The default batch size is 32, and the GPU memory usage is 3~4 GB for Resnet50. Did you run the code in the docker? If so, you should check the share memory of your docker container.

[lzgahbb]

Thanks, I solved this problem by retring your code in a new directory.

[saxg]

I have faced the problem same as that you said, what is the meaning of the "in a new directoty"