CudaHostAlloc takes a lot of time during training

[niuliling123]

🐛 Describe the bug

set: pin_memory =True in Dataloader

Versions

pytorch 1.13
numpy 1.21
python3.8

cc @ssnl @VitalyFedyunin @ejguan @dzhulgakov @ptrblck

[xw285cornell]

this CUDA API is supposed to be very slow. There is cuda host caching allocator that will creates a memory pool to avoid calling this API. Can you make sure you run the data loader multiple times to warm up the cache? It should stop showing up after a while

[niuliling123]

I have already used data cache to speed up. it is very common in my model.
What is the size of pin_memory ？During the training process, I saw that there was still a large amount of remaining GPU memory.

[gokulavasan]

@niuliling123 What is the approx size of batch (received by dataloader from the worker) after collation?

[niuliling123]

batch_size is 16