How to reduce GPU usage ?

[Mr-LiuDC]

GPU 1050Ti with 4G memory, got this error.

RuntimeError: CUDA out of memory. Tried to allocate 768.00 MiB (GPU 0; 4.00 GiB total capacity; 2.71 GiB already allocated; 0 bytes free; 2.81 GiB reserved in total by PyTorch)

[mraduldubey]

OutOfMemoryError: CUDA out of memory. Tried to allocate 1024.00 MiB. GPU 0 has a total capacty of 6.00 GiB of which 0 bytes is free. Of the allocated memory 4.60 GiB is allocated by PyTorch, and 129.79 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

GPU: NVIDIA GeForce RTX 3060 Laptop GPU

Kinda surprising. This is with VIT_H model.

I do print print(torch.cuda.memory_summary(device=None, abbreviated=False)) before and after the culprint lines:

sam.to(device='cuda')
predictor = SamPredictor(sam)

Before these lines, GPU memory is almost 0. And after these lines, I get OOM error. So, I suppose VIT_H cant be run with 6GB VRAM ? I will try with VIT_L model.

EDIT

Switched to VIT_L model for a successful inference.

[chuanheliu]

generator = SamAutomaticMaskGenerator(sam, output_mode="binary_mask")

change points_per_batch=16 or less

generator = SamAutomaticMaskGenerator(sam, points_per_batch=16, output_mode="binary_mask")