CUDA 12 Support #1201

quasiben · 2023-06-22T16:58:54Z

Adds CUDA 12 Support to build matrix

Fixes #1115

jakirkham · 2023-06-22T17:44:22Z

Thanks Ben! 🙏

Would suggest looking at the RMM CUDA 12 PR ( rapidsai/rmm#1223 ) and lining this up with that

dependencies.yaml

jakirkham · 2023-06-22T19:07:07Z

Think we may also want these GHA changes

Not seeing any environment files here

Also pyproject.toml, which the Conda recipe pulls from, doesn't appear to have any CUDA specific changes needed

So that may be it

bdice · 2023-06-22T19:45:59Z

I agree with all @jakirkham's suggestions above. Maybe also need updates for docs/source/install.rst?

jakirkham · 2023-06-22T19:51:49Z

Thanks Bradley! 🙏

Good catch. Yeah this cudatoolkit here should be changed to cuda-version

https://github.com/rapidsai/dask-cuda/blame/83c64765ec6612cb7c04b077e30edf6f846d90e0/docs/source/install.rst#L19

pentschev · 2023-06-23T14:34:14Z

I pushed GHA and docs changes.

@jakirkham @bdice could you check that the docs is correct, is that supposed to work for both CUDA 11.x and 12.x, or the correct way to document it is CUDA 11.x requires cudatoolkit=11.x and CUDA >= 12.x requires cuda-verison=12.x?

pentschev · 2023-06-23T16:42:41Z

Also seems like cuCIM needs to be handled before Dask-CUDA.

jakirkham · 2023-06-23T18:06:15Z

cuda-version should work on both CUDA 11 & 12

Ok could you please add cuCIM to the list in this issue ( #1115 )?

…port

pentschev · 2023-06-27T14:27:11Z

I'm not quite sure what's going on in the CUDA 12 tests, it seems Numba is failing to find NVVM files:

Traceback (most recent call last):
  File "<string>", line 7, in <module>
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/runtime.py", line 111, in get_version
    self.cudaRuntimeGetVersion(ctypes.byref(rtver))
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/runtime.py", line 65, in __getattr__
    self._initialize()
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/runtime.py", line 51, in _initialize
    self.lib = open_cudalib('cudart')
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/libs.py", line 63, in open_cudalib
    path = get_cudalib(lib)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cudadrv/libs.py", line 55, in get_cudalib
    libdir = get_cuda_paths()[dir_type].info
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cuda_paths.py", line 223, in get_cuda_paths
    'nvvm': _get_nvvm_path(),
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/cuda/cuda_paths.py", line 201, in _get_nvvm_path
    candidates = find_lib('nvvm', path)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/misc/findlib.py", line 44, in find_lib
    return find_file(regex, libdir)
  File "/opt/conda/envs/test/lib/python3.10/site-packages/numba/misc/findlib.py", line 56, in find_file
    entries = os.listdir(ldir)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/envs/test/nvvm/lib64'

@bdice @jakirkham do you know if we're missing something here? Also cc @gmarkall for Numba, in case you have suggestions.

jakirkham · 2023-06-27T16:14:06Z

Yeah numba needs cuda-nvcc installed

This is needed to get nvvm, cudart & compilers

pentschev · 2023-06-27T19:10:50Z

Yeah numba needs cuda-nvcc installed

This is needed to get nvvm, cudart & compilers

Shouldn't this be provided when Numba gets installed then? Note that it happens during a test, and I don't think Dask-CUDA should be requiring cuda-nvcc at runtime.

bdice

Requesting a small change to use cuda-version in dependencies.yaml.

bdice · 2023-06-27T19:15:23Z

dependencies.yaml

@@ -59,6 +59,10 @@ dependencies:
              cuda: "11.8"
            packages:
              - cudatoolkit=11.8


Can we change all these entries like the example below? This is needed to constrain the CUDA version of the RAPIDS packages used for Python testing. (It may be possible to derive the correct constraint from a pin on cudatoolkit but that's going to be less direct for the environment solver.)

Suggested change

- cudatoolkit=11.8

- cuda-version=11.8

- cudatoolkit

jakirkham · 2023-06-27T19:20:22Z

Similar to ucx, which can operate without cudatoolkit, numba doesn't require cuda-nvcc (except when CUDA is used). So it is up to us to install this when we need it

Edit: Maybe we can handle this on the cuDF side

pentschev · 2023-06-27T19:48:15Z

Similar to ucx, which can operate without cudatoolkit, numba doesn't require cuda-nvcc (except when CUDA is used). So it is up to us to install this when we need it

Edit: Maybe we can handle this on the cuDF side

Numba is an optional dependency, so the user will certainly need to satisfy that when using Numba+Dask-CUDA but without RAPIDS. I'm guessing we will want to update Numba's installation instructions for CUDA 12 too?

jakirkham · 2023-06-27T19:58:13Z

Yeah that's a good idea. Could you please raise an issue on the Numba repo and cc me?

pentschev · 2023-06-27T20:16:05Z

@jakirkham raised the issue in numba/numba#9045

pentschev · 2023-06-27T20:46:36Z

Yeah numba needs cuda-nvcc installed

This is needed to get nvvm, cudart & compilers

Adding cuda-nvcc raised a different error now:

cuda-nvcc: Please add the `compiler("c")` and `compiler("cxx")` packages to the environment.

It seems that this can't be added as a runtime dependency, is that the case?

jakirkham · 2023-06-27T21:48:05Z

Thanks Peter! 🙏

This could be worked around by adding the c-compiler & cxx-compiler packages. Alternatively cuda-compiler pulls both of them in for us

That said, it seems reasonable to harden this requirement in cuda-nvcc. Submitted upstream PR ( conda-forge/cuda-nvcc-feedstock#21 )

pentschev · 2023-06-28T08:42:52Z

Thanks @jakirkham , after merging of conda-forge/cuda-nvcc-feedstock#21 all tests now pass! 😄

bdice · 2023-06-28T14:37:17Z

dependencies.yaml

+              - cudatoolkit
+          - matrix:
+              cuda: "12.0"
+            packages:


Do we need cuda-nvrtc for Numba?

You can get away without it if you're not linking .cu files with Python kernels or using float16. But for full functionality it should be present.

Added this suggestion below

jakirkham

Thanks Peter! 🙏

This is looking pretty good. Added a minor suggestion to include cuda-nvrtc based on the thread below

jakirkham · 2023-06-28T17:32:58Z

dependencies.yaml

+              - cudatoolkit
+          - matrix:
+              cuda: "12.0"
+            packages:


Added this suggestion below

jakirkham · 2023-06-28T17:33:02Z

dependencies.yaml

+              cuda: "12.0"
+            packages:
+              - cuda-version=12.0
+              - cuda-nvcc


Suggested change

- cuda-nvcc

- cuda-nvrtc

From what I understand of @gmarkall 's suggestion we don't need cuda-nvrtc given we're not doing any linkage whatsoever in Dask-CUDA, thus I think we should have only cuda-nvcc here.

Ok will leave this here and we can revisit as needed

jakirkham · 2023-06-28T20:34:32Z

/merge

jakirkham · 2023-06-28T20:34:49Z

Thanks all! 🙏

pentschev · 2023-06-28T20:35:40Z

Thanks everyone for reviews! 😄

add cuda 12 to matrix

415cc03

quasiben requested a review from a team as a code owner June 22, 2023 16:58

use cuda-version for cuda 12

1686d6f

quasiben commented Jun 22, 2023

View reviewed changes

dependencies.yaml Show resolved Hide resolved

quasiben added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jun 22, 2023

pentschev added 2 commits June 23, 2023 07:25

Update GHA to cuda-120 branch

99017fa

Update installation docs

6dfedd7

Merge remote-tracking branch 'upstream/branch-23.08' into cuda-12-sup…

bfb8739

…port

bdice requested changes Jun 27, 2023

View reviewed changes

pentschev added 2 commits June 27, 2023 12:50

Add missing cuda-version constraint to dependencies.yaml

5f76e0d

Add cuda-nvcc to satisfy Numba's CUDA dependencies

9fa0976

gmarkall mentioned this pull request Jun 28, 2023

Docs should be updated for CUDA 12 numba/numba#9045

Closed

bdice reviewed Jun 28, 2023

View reviewed changes

bdice approved these changes Jun 28, 2023

View reviewed changes

jakirkham reviewed Jun 28, 2023

View reviewed changes

raydouglass approved these changes Jun 28, 2023

View reviewed changes

jakirkham changed the title ~~[WIP] CUDA 12 Support~~ CUDA 12 Support Jun 28, 2023

jakirkham approved these changes Jun 28, 2023

View reviewed changes

rapids-bot bot merged commit eafde5f into rapidsai:branch-23.08 Jun 28, 2023
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA 12 Support #1201

CUDA 12 Support #1201

quasiben commented Jun 22, 2023 •

edited by jakirkham

jakirkham commented Jun 22, 2023

jakirkham commented Jun 22, 2023

bdice commented Jun 22, 2023

jakirkham commented Jun 22, 2023

pentschev commented Jun 23, 2023

pentschev commented Jun 23, 2023

jakirkham commented Jun 23, 2023

pentschev commented Jun 27, 2023

jakirkham commented Jun 27, 2023

pentschev commented Jun 27, 2023

bdice left a comment

bdice Jun 27, 2023

jakirkham commented Jun 27, 2023 •

edited

pentschev commented Jun 27, 2023

jakirkham commented Jun 27, 2023

pentschev commented Jun 27, 2023

pentschev commented Jun 27, 2023

jakirkham commented Jun 27, 2023

pentschev commented Jun 28, 2023

bdice Jun 28, 2023

gmarkall Jun 28, 2023 •

edited

jakirkham Jun 28, 2023

jakirkham left a comment

jakirkham Jun 28, 2023

jakirkham Jun 28, 2023

pentschev Jun 28, 2023

jakirkham Jun 28, 2023

jakirkham commented Jun 28, 2023

jakirkham commented Jun 28, 2023

pentschev commented Jun 28, 2023

CUDA 12 Support #1201

CUDA 12 Support #1201

Conversation

quasiben commented Jun 22, 2023 • edited by jakirkham

jakirkham commented Jun 22, 2023

jakirkham commented Jun 22, 2023

bdice commented Jun 22, 2023

jakirkham commented Jun 22, 2023

pentschev commented Jun 23, 2023

pentschev commented Jun 23, 2023

jakirkham commented Jun 23, 2023

pentschev commented Jun 27, 2023

jakirkham commented Jun 27, 2023

pentschev commented Jun 27, 2023

bdice left a comment

Choose a reason for hiding this comment

bdice Jun 27, 2023

Choose a reason for hiding this comment

jakirkham commented Jun 27, 2023 • edited

pentschev commented Jun 27, 2023

jakirkham commented Jun 27, 2023

pentschev commented Jun 27, 2023

pentschev commented Jun 27, 2023

jakirkham commented Jun 27, 2023

pentschev commented Jun 28, 2023

bdice Jun 28, 2023

Choose a reason for hiding this comment

gmarkall Jun 28, 2023 • edited

Choose a reason for hiding this comment

jakirkham Jun 28, 2023

Choose a reason for hiding this comment

jakirkham left a comment

Choose a reason for hiding this comment

jakirkham Jun 28, 2023

Choose a reason for hiding this comment

jakirkham Jun 28, 2023

Choose a reason for hiding this comment

pentschev Jun 28, 2023

Choose a reason for hiding this comment

jakirkham Jun 28, 2023

Choose a reason for hiding this comment

jakirkham commented Jun 28, 2023

jakirkham commented Jun 28, 2023

pentschev commented Jun 28, 2023

quasiben commented Jun 22, 2023 •

edited by jakirkham

jakirkham commented Jun 27, 2023 •

edited

gmarkall Jun 28, 2023 •

edited