Support CUDA pinned memory in DataLoader

[colesbury]

CUDA pinned memory is important for efficient execution because it allows for faster data transfers and non-blocking CUDA copies.

The copy from normal memory to pinned memory can take significant time. A batch of 256x3x224x224 FloatTensor takes about 110ms on my computer to copy. Currently we can only do the copy on the main process because inter-process shared Tensor/Storages are copied to non-page locked shared memory. For small conv nets on fast GPUs, we probably need to do the copy in the background.

I believe we can page-lock the shared memory via cudaHostRegister. We would probably need to unregister it via cudaHostUnregister before freeing the memory.

This would require some knowledge of CUDA in the shared memory code or at least a free hooks to call cudaHostUnregister.

[apaszke]

Hm, this is tricky. I think cudaHostRegister not only does the syscalls that page lock the memory, but it also registers it in its own data structures (it doesn't treat manually locked memory as pinned). I don't think there's any way to transfer this, especially that different processes might unlock it, while it's still being used somewhere else. I'd be for closing this issue, it's pretty complex.

[colesbury]

I think we need a solution to this problem even if we don't do it in a separate process. We could probably do a copy to pinned memory in the main process in a thread which releases the GIL.

cudaHostRegister has a cudaHostRegisterPortable flag which allows the memory to be used by other cuda contexts. I had assumed this meant other processes as well. It sounds like someone got it to work in the forum post below:

https://devtalk.nvidia.com/default/topic/510439/share-gpu-host-pinned-memory-between-host-processes/?offset=6

[apaszke]

I can't find any solution for the host memory sharing in the thread you liked, they only solved the device memory sharing. We should probably ask how cudaHostRegisterPortable behaves if one of the processes frees the memory, but another one is still using it.

On the other hand, if you could get a big pinned buffer mapped as shared memory, you could slice it and send the chunks to worker processes that would fill them. Once you move your tensor to shared memory, it will be the same for it and its slices until the program terminates (or it gets freed).

[colesbury]

Oh I pasted the wrong link. Here it is:
https://devtalk.nvidia.com/default/topic/494915/sharing-cuda-host-memory-between-processes/

[apaszke]

Still, we should probably ask someone from NVIDIA what happens if we register it in e.g. 3 processes and then unregister it in one of them - is it no longer page-locked in the other 2?

[colesbury]

Oh I pasted the wrong link. Here it is:
https://devtalk.nvidia.com/default/topic/494915/sharing-
cuda-host-memory-between-processes/

On Sunday, October 23, 2016, Adam Paszke <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

I can't find any solution for the host memory sharing in the thread you
liked, they only solved the device memory sharing. We should probably ask
how cudaHostRegisterPortable behaves if one of the processes frees the
memory, but another one is still using it.

On the other hand, if you could get a big pinned buffer mapped as shared
memory, you could slice it and send the chunks to worker processes that
would fill it. Once you move your tensor to shared memory, it will be the
same for it and its slices until the program terminates (or it gets freed).

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#139 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAoB-ixxgK2DjsQLljASTjXe07lFx-eDks5q29f_gaJpZM4KaE8y
.

[apaszke]

@ngimel @thatguymike Could you please take a look at this issue? Can you share pinned memory between processes? Do you know what happens if one of them unpins/frees the memory?

[ngimel]

@colesbury, there's nothing special about copying to pinned memory, it should not be going at 3 GB/s on any reasonable CPU. What I think is happening is host mem bandwidth is depleted by image transformations in the different threads. For each image, data augmentation (ColorJitter, Lighting, Normalize etc) is >=10 image reads/writes, and all that consumes host mem bandwidth. Offloading image transformations to GPU might help.
@apaszke, I don't know right away, I'll ask.

[ngimel]

@apaszke You can share pinned memory between processes using shared memory segments. Each process has to cudaHostRegister this memory to be able to use it as pinned (and cudaHostUnregister it). We are doing this e.g. in multi-process version of nccl, you can take a look at https://github.com/NVIDIA/nccl/blob/master/src/core.cu

[apaszke]

@ngimel So just to confirm after such sequence:

1. A registers some pages
2. B registers the same pages
3. A unregisters them

the pages are still locked and B can continue using them, right?

[ngimel]

@apaszke, sorry, did not see your question in time. Yes, that should work.