no difference in memory usage

[ofrimasad]

Hi.
I am training my network with bnb.optim.Adam8bit vs torch.optim.Adam but I don't see any difference in memory consumption.

Running on GTX 2080Ti (single gpu or DDP).
with cudatoolkit 11.1.74
bitsandbytes-cuda111

looking in nvidia-smi I see 9.6GB in both cases
Am I missing something here?

[JoeyQQ]

hi there, i got the same error.
did you solve this issues?

[LeeDoYup]

me too

[TimDettmers]

If the network has few parameters you will not only small memory reductions (for example for ResNet 50).

The other problem is that nvidia-smi only shows the maximum memory allocated. PyTorch keeps unallocated memory around to made allocation more efficient. So it may seem that the memory is not decreased while it actually is.