CUDA error when trying to use multiple GPUs: nnconv_forward_blas: CUDA error [cuda: forward: pointer does not correspond to a registered memory region]

[junaidmalik09]

I'm gettng a CUDA memory error when trying to run my FCN code on more than 1 GPUs. As soon as the training finishes in the first epoch and validation is about to start, I get the following

the error is trailed by a string of "cudaHostUnregister failied" warnings as shown below:

Has anyone faced a similar issue? If yes, any help would be much appreciated. Thanks!

[junaidmalik09]

apparently, this only happens when using Matconvnet 23 (latest). I switched back to matconvnet 17 which I had used earlier and the error disappears

[nicjac]

I got the same issue. Any pointer as to how to fix this?

[jotaf98]

A couple of things to try:

1. The multi-GPU code has changed a bit recently; do you still have the same issue with Matconvnet 24?
2. Try switching between opts.parameterServer.method = 'mmap' and 'tmove'

[zoraZHJ]

on windows, opts.parameterServer.method = 'mmap'
and on linux Ubuntu, set to 'tmove'

[Mirsadeghi]

I got the same issue! Any solution?