[NVPTX] bad binary since cuda-11.3

[ye-luo]

My application fails with wrong numbers when using cuda >= 11.3 toolchain. Checked up to 11.6
source and assembly code
badcubin.zip

psi_list_ptr[iw] = psi_local;

both sides are std::complex. Bad binary caused the imaginary part of the left hand side has value 0.

--save-temps assembly files from CUDA 11.2 and 11.3, they differ only by

diff cuda11.3/MultiSlaterDetTableMethod-openmp-nvptx64-nvidia-cuda.s cuda11.2/
5c5
< .version 7.3
---
> .version 7.2

If I compile the whole application with CUDA 11.3 toolchain, test fails. Since my application is OpenMP offload, the nvptx pass invokes ptxas, If I use ptxas from CUDA 11.2 to generate cubin for the failing file and all the rest uses CUDA 11.3. my test passes.
So my guess is the nvptx backend and ptxas (>=7.3) have some incompatibility and caused bad binary. I just leave my analysis here, hopefully backend experts will have more ideas.

Q: Is there a way to force clang to generate assembly files with a different PTX version? In this way combined with --ptxas-path, I can use an alternative ptxas while the rest remains with the primary CUDA toolkit I need to use.

[Artem-B]

Is there a way to force clang to generate assembly files with a different PTX version?

I do not think that's not going to help you. The issue is apparently with ptxas. PTX version itself is inconsequential.
Using a different CUDA version (and thus different ptxas) or explicitly specified ptxas from a different CUDA version are your only practical options.

NVIDIA almost never fixes bugs in released CUDA versions, so, if the bug is not fixed in CUDA-11.6.2, you will need to wait for a new CUDA release and hope that they fix it there.

If you could create a small reproducer that could be used as a test one could compile and run, that would help me to file a bug with NVIDIA and get it fixed. Ideally a pure CUDA source, without having to use openmp. A reasonably small PTX source which assembles to something obviously wrong on the SASS level would work, too.

[ye-luo]

@Artem-B thank you for the comment. Yes as a workaround, I'd like to explicitly specify ptxas from a different CUDA version, namely point to the 11.2 one while using 11.4 with all the rest. However right now if Clang recogonize 11.4, it outputs the assembly file with ptx version 7.4 and ptxas from 11.2 rejects it saying 7.4 is not supported. So If I can force outputing the PTX assembly files with 7.2 version label, I can implement a workaround for now forcing ptxas from 11.2 and using CUDA 11.4 in all the rest.

Not much luck when I tried to create a small reproducer. Will try SASS.

[Artem-B]

You could try overriding the PTX version with -Xclang -target-feature -Xclang -ptx74 -Xclang -target-feature -Xclang +ptx72.
It would need to be tweaked if you use CUDA and ptxas versions other than 11.4 and 11.2.

If you use a newer ptxas extra options should not be needed, but the produced cubin may not be compatible with the older CUDA runtime.

[Artem-B]

Would you be able to reduce the reproducer? It would be great to figure out what triggers the issue and, if it's indeed a miscompile in ptxas, report it to NVIDIA so they can get it fixed.

[markdewing]

This issue affects complex number types and structures.
I'm trying to create a smaller reproducer for this issue, and so far have not been successful.
The failure mode is the assignment of the reduction variable after the reduction fails to copy the second item (for the type with 3 elements, the third element copies okay).
What is important is the reduction. (That is, if the pragma with the reduction is removed, the code works)

A snippet of code with the issue

#pragma omp target teams distribute is_device_ptr(psiV_list_devptr, psiMinv_temp_list_devptr) \
                              map(always, from:curRatio_list_ptr[:nw])
      for (size_t iw = 0; iw < nw; iw++)
      {
        GradType ratioGradRef_local(0);
#pragma omp parallel for reduction(+ : ratioGradRef_local)
        for (size_t i = 0; i < NumPtcls; i++)
        {
          const size_t J            = confgListOccup_ptr[i];
          psiV_temp_list_ptr[iw][i] = psiV_list_devptr[iw][J];
          ratioGradRef_local += psiMinv_temp_list_devptr[iw][i * psiMinv_cols + WorkingIndex] * dpsiV_list_ptr[iw][J];
        }
  
        // Workaround for https://github.com/QMCPACK/qmcpack/issues/4767
        // Compiler-generated assignment fails for the second component.
        // Workaround is to assign each component individually.
        // This assignment fails
        //ratioGradRef_list_ptr[iw] = ratioGradRef_local;
        // Assigning individual components works
        ratioGradRef_list_ptr[iw][0] = ratioGradRef_local[0];
        ratioGradRef_list_ptr[iw][1] = ratioGradRef_local[1];
        ratioGradRef_list_ptr[iw][2] = ratioGradRef_local[2];
        ...

GradType is basically a struct with 3 doubles.
More precisely, it is of type TinyVector<double, 3>, where TinyVector is defined as

template<class T, unsigned D>
struct TinyVector
{
  using Type_t = T;
  enum
  {
    Size = D
  };
  T X[Size];
  ...

And, as originally posted, the assignment succeeds with CUDA 11.2.2, and fails with any later version of CUDA.

The issue can be seen by adding printf's immediately after the assignment

      ratioGradRef_list_ptr[iw] = ratioGradRef_local;
      printf("iw = %lu ratioGradRef_local = %g %g %g\n",iw,ratioGradRef_local[0],ratioGradRef_local[1],ratioGradRef_loc  al[2]);
      printf("iw = %lu ratioGradRef_list = %g %g %g\n",iw,ratioGradRef_list_ptr[iw][0],ratioGradRef_list_ptr[iw][1],rat  ioGradRef_list_ptr[iw][2]);

Will produce the output

iw = 0 ratioGradRef_local = 4.64154 3.48115 2.18973
iw = 1 ratioGradRef_local = 0.701702 0.801945 0.443089
iw = 0 ratioGradRef_list = 4.64154 0 2.18973
iw = 1 ratioGradRef_list = 0.701702 0 0.443089

[markdewing]

The type of the index in the outer loop over nw seems to matter. If the type is changed from size_t to uint32_t, the problem seems to go away. Still can't reproduce in a smaller, standalone test, though.