Data transfer in collate v.s. during training loop

[bowenxieoregon]

Hi all,

On page 224, when talking about the device setting, the book says: Having this as part of the collate function offers the advantage of performing this device transfer process as a background process outside the training loop, preventing it from blocking the GPU during model training.

I do not understand this statement, so I asked ChatGPT, and it tells me that performing device transferring during training loop is preferred actually if we use .to(device, non_blocking = True) during training loop. And if you perform device transferring in collate, it can only work for num_workers = 0, single GPU.

Anyone has any ideas on the discussion above?