GPU memory not fixed when training

[ProBURN-E]

Thank you for your great work！
Why does the GPU memory change all the time during training instead of being fixed all the time?
Except for changing scale to 2, I use the default configs.

[neosr-project]

Hi @ProBURN-E, thanks. VRAM usage varies due to multiple factors, it's normal. There are internal bottlenecks that causes it, as well as different network designs that influences it. For example, networks like as compact and span have a stable gpu-util and vram usage, as opposed to omnisr hich has more fluctuations.
Another reason could be slow dataloading times. If you have an SSD, it is recommended that you put your dataset in it, as it will avoid high reading times from HDD.

ps: another factor could be the image resolution. If your dataset has big images, it will take more time to decode, so it creates a bottleneck. It is recommended that you tile your images to a lower resolution, such as 512x512px.