8 GPUs got an error, but 6 GPUs is good.

[swzhang5]

Thanks for sharing your great work.
But when I tried to retrain your work with 8GPUs, I got an error: CUDA error: device-side assert triggered.
However, 6GPUs can work, but the mAP is not good. I don't know the reason, do you know how to make it work under 8GPUs? Or could you give me a config for 6GPUs? Thank you

[jason718]

we never tested 6 gpu setting, sorry.

try re-run the program with 8 gpu again? I also run into the same issue sometimes, even though not so often.
see known issue 1

[bradezard131]

My guess is the BCE Epsilon is too small for the WSDDN loss. I find in my own code that I need 1e-8 on the clamp to avoid device-side asserts. You could try running with the environment variable CUDA_LAUNCH_BLOCKING=1 set, which may slow down training but will pin down the error to an exact line of code.

[jason718]

My guess is the BCE Epsilon is too small for the WSDDN loss. I find in my own code that I need 1e-8 on the clamp to avoid device-side asserts. You could try running with the environment variable CUDA_LAUNCH_BLOCKING=1 set, which may slow down training but will pin down the error to an exact line of code.

Thanks for sharing~ Very interesting. I took that clamping threshold from OICR and never changed it. I'll also try it here.

[swzhang5]

My guess is the BCE Epsilon is too small for the WSDDN loss. I find in my own code that I need 1e-8 on the clamp to avoid device-side asserts. You could try running with the environment variable CUDA_LAUNCH_BLOCKING=1 set, which may slow down training but will pin down the error to an exact line of code.

Thanks! The program is working well under 8GPUs now

[jason718]

Awesome. Closing this now, and feel free to re-open.