Fix a potential race condition in the CUDA backend

[Rombur]

There is a potential race condition when using multiple Teams parallel_for/reduce with the same instance. It's possible that the scratch memory pointer gets invalidated before the kernel is done. In HIP this was fixed by using a guard lock but that does not work for CUDA because we need ParallelFor/Reduce to have a copy constructor. Instead, I have created a pool of 10 pointers (I just picked a number, I don't have a good reason to choose 10) and we cycle through them when we need one. I know that there isn't any test yet, but I am working on that.

[masterleinad]

What's the reason that ParallelFor needs to be copy-constructible for Cuda but not for HIP? I guess the backends should be consistent in this regard.

[crtrott]

This is a race condition only with multiple threads dispatching work right?

[Rombur]

What's the reason that ParallelFor needs to be copy-constructible for Cuda but not for HIP? I guess the backends should be consistent in this regard.

No, the backends should not be consistent. The reason we don't use a copy in HIP is because the compiler sometimes creates a bugged kernel when the kernel is copied. So we needed to create a work around which has performance implication.

This is a race condition only with multiple threads dispatching work right?

Correct.

[crtrott]

Ok I get it. This works because the race is only on the reallocation/launch of the kernel. If you actually deallocate something and another kernel already in flight is using the same scratch you are good anyway since the realloc will block the device. Hence releasing the lock at the end of ParalleFor (which is NOT the end of the asynchronous kernel) is fine.