Installation issue: gromacs fail due to cuda compute_30 (dropped in cuda11)

[michaelkarlcoleman]

Omitting a lot, as I believe nvcc from cuda-11.x simply will not generate compute_30 code, thus the error message.

See http://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

Default solution is to just remove compute_30. More complex would be to allow different CUDA versions and adjust the generated code to match.

Steps to reproduce the issue

$ spack install gromacs +cuda

     474    /gpfs/packages/spack/spack/opt/spack/linux-rhel7-broadwell/gcc-8.2.0/cuda-11.0.2-tlog2jv6ze6lfxk47omkfabtz7474gbw/bin/nvcc -M -D__CUDACC__ /tmp/swmgr/s
            pack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/gromacs/nbnxm/cuda/nbnxm_cuda.cu -o /tmp/swmgr/spack-stage/spack-s
            tage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-build/src/gromacs/CMakeFiles/libgromacs.dir/nbnxm/cuda/libgromacs_generated_nbnxm_cuda.cu.o.
            NVCC-depend -ccbin /gpfs/packages/spack/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/gcc-8.2.0-kot2sql3i2pckkfopvmxdmbdopuwy42t/bin/g++ -m64 -Dlibgroma
            cs_EXPORTS -DHAVE_CONFIG_H -DGMX_DOUBLE=0 -DUSE_STD_INTTYPES_H -Xcompiler ,\"-mavx2\",\"-mfma\",\"-pthread\",\"-fPIC\",\"-O2\",\"-g\",\"-DNDEBUG\",\"-f
            excess-precision=fast\",\"-funroll-all-loops\" -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm
            _37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -ge
            ncode arch=compute_70,code=sm_70 -gencode arch=compute_35,code=compute_35 -gencode arch=compute_50,code=compute_50 -gencode arch=compute_52,code=comput
            e_52 -gencode arch=compute_60,code=compute_60 -gencode arch=compute_61,code=compute_61 -gencode arch=compute_70,code=compute_70 -gencode arch=compute_7
            5,code=compute_75 -use_fast_math -DNVCC -I/gpfs/packages/spack/spack/opt/spack/linux-rhel7-broadwell/gcc-8.2.0/cuda-11.0.2-tlog2jv6ze6lfxk47omkfabtz747
            4gbw/include -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/external/thread_mpi/include -I/tmp/swmg
            r/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-build/src -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7
            untnzai2yrdnef7nklow2mc/spack-src/src -I/gpfs/packages/spack/spack/opt/spack/linux-rhel7-broadwell/gcc-8.2.0/openmpi-3.1.6-tg6m4mmn2p3vxbdpopsqgfesihlm
            3g5f/include -I/gpfs/packages/spack/spack/opt/spack/linux-rhel7-broadwell/gcc-8.2.0/hwloc-1.11.11-s54td4k4foxz3kdukoesox46risjltch/include -I/usr/inclu
            de -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/external -I/tmp/swmgr/spack-stage/spack-stage-gro
            macs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/external/tng_io/include -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai
            2yrdnef7nklow2mc/spack-build/tng/include -I/tmp/swmgr/spack-stage/spack-stage-gromacs-2020.3-2lg7rupv7untnzai2yrdnef7nklow2mc/spack-src/src/external/lm
            fit
     475    nvcc fatal   : Unsupported gpu architecture 'compute_30'

Information on your system

Additional information

• spack-build-out.txt
• spack-build-env.txt

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have run spack maintainers <name-of-the-package> and @mentioned any maintainers
• I have uploaded the build log and environment files
• I have searched the issues of this repo and believe this is not a duplicate

[michaelkarlcoleman]

This seems to be coming from gromacs upstream:

./gromacs-2020.3/cmake/gmxManageNvccConfig.cmake:    list (APPEND GMX_CUDA_NVCC_GENCODE_FLAGS "-gencode;arch=compute_30,code=sm_30")

Looks like they haven't caught up to CUDA 11. One solution would be to simply remove this line (via patch) if CUDA version is 11+.

[adamjstewart]

@junghans @marvinbernhardt

[junghans]

/cc @mabraham @pszi1ard @eirrgang

[mabraham]

Upstream issue at https://gitlab.com/gromacs/gromacs/-/issues/3632

[mabraham]

Changes proposed at https://gitlab.com/gromacs/gromacs/-/merge_requests/461 would likely work in the meantime (and similar would be needed for earlier GROMACS release branches, although those are no longer supported upstream).

[pszi1ard]

Looks like they haven't caught up to CUDA 11.

Not yet, but on the other hand, CUDA 11 seems to have deprecated compute capability 3.0 overnight (?).

One solution would be to simply remove this line (via patch) if CUDA version is 11+.

For <=v2020.4 that is a reasonable workaround; alternatively, the spack config could explicitly override the CUDA targets on the CMake invocation, e.g. -DGMX_CUDA_TARGET_SM="35;37;52;61;60;70;75" -DGMX_CUDA_TARGET_COMPUTE="70" (could consider adding Ampere too, i.e. 80).

[michaelkarlcoleman]

We have mostly K80s here (but a few newer cards). Would be quite handy if this worked by default, or at least with some extra '+' flag or something.

[pszi1ard]

We have mostly K80s here (but a few newer cards). Would be quite handy if this worked by default, or at least with some extra '+' flag or something.

Not sure what do you mean by '+' flag, can you elaborate? As noted above you can pass those extra CMake flags at configure-time and you'll get essentially the same set of flags (and behavor) as the upstream fix implements.

[michaelkarlcoleman]

I'm not exactly sure what I mean either, but I had in mind a "spack-ish" syntax to specify which cards to support, if that seems necessary. Just supporting all possible cards would be great as well, and maybe doesn't need a flag? An additional possibility, though more work, would be to move all of this decision-making/flags to the 'cuda' spack module. Then spack packages that depend on that module could be taught to do what it said, rather than trying to guess the right values themselves. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Tuesday, August 25, 2020 10:02 AM, Szilárd Páll ***@***.***> wrote: > We have mostly K80s here (but a few newer cards). Would be quite handy if this worked by default, or at least with some extra '+' flag or something. Not sure what do you mean by '+' flag, can you elaborate? As noted above you can pass those extra CMake flags at configure-time and you'll get essentially the same set of flags (and behavor) as the upstream fix implements. — You are receiving this because you authored the thread. Reply to this email directly, [view it on GitHub](#18239 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAABIAEASQSHH3Q3GBBBLTLSCPVDHANCNFSM4QJXJZEQ).

[pszi1ard]

I'm not exactly sure what I mean either, but I had in mind a "spack-ish" syntax to specify which cards to support, if that seems necessary.

I can unfortunately not help with that.

Just supporting all possible cards would be great as well, and maybe doesn't need a flag?

The GROMACS build system already does that. In this case that mechanism does not work well enough because a GPU architecture was deprecated without advance notice and the GROMACS build system is not yet adapted to the CUDA 11 limitations.

[mabraham]

Changes proposed at https://gitlab.com/gromacs/gromacs/-/merge_requests/461 would likely work in the meantime (and similar would be needed for earlier GROMACS release branches, although those are no longer supported upstream).

Fix is merged and will be included in 2020.4

[victoryang00]

For a temporary solution: you can merely download their upstream release from here that fixed the cuda-11 not compatible issue. and put them to /path/to/spack/var/spack/cache/_source-cache/archive/cd/cd12bd4977e19533c1ea189f1c069913c86238a8a227c5a458140b865e6e3dc5.tar.gz.

The checksum can be gotten by sha256sum gromacs-prepare-2020.4.tar.gz. Besides, you should modify the 2020.4 checksum in /path/to/spack/var/spack/repos/builtin/packages/gromacs/package.py, add version('2020.4', sha256='cd12bd4977e19533c1ea189f1c069913c86238a8a227c5a458140b865e6e3dc5').

Then you can simply run spack install gromacs@2020.4+mpi+cuda

[roguephysicist]

@victoryang00, your link leads to a "404 Git Resource Not found".

[junghans]

2020.4 will be released soon, so let's just wait.