cuml: Build CUDA 12 packages #5318

vyasr · 2023-03-31T21:40:54Z

This PR builds libcuml and cuml conda packages using CUDA 12. Resolves #5234.

vyasr · 2023-03-31T21:58:53Z

This PR needs to be updated to pull artifacts from rapidsai/raft#1388 once those are ready, and it needs a cumlprims PR as well.

dantegd · 2023-07-11T05:26:32Z

There was one pytest failure that seems relevant and we need to investigate it but it's not a build blocker to get packages ready. Opened issue #5497 and skipped the pytest temporarily so we can get CUDA 12 packages early for testing.

dantegd · 2023-07-11T13:52:00Z

Follow up comment about the prior failed pytest, it happened to be an OOM orthogonal to CUDA 12. @viclafargue submitted a fix in #5498 and I merged the fix into this PR as well.

bdice

Awesome progress. Comments attached.

dependencies.yaml

conda/recipes/cuml/conda_build_config.yaml

conda/recipes/libcuml/conda_build_config.yaml

conda/recipes/libcuml/meta.yaml

dependencies.yaml

python/cuml/tests/test_pca.py

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

vyasr

This is getting super close!

conda/environments/all_cuda-120_arch-x86_64.yaml

conda/recipes/libcuml/meta.yaml

vyasr · 2023-07-11T23:49:54Z

conda/recipes/cuml/meta.yaml

+    {% if cuda_major == "11" %}
+    - cuda-python ==11.7.1
+    - cudatoolkit
+    - libcublas {{ cuda11_libcublas_host_version }}
+    - libcublas-dev {{ cuda11_libcublas_host_version }}
+    - libcurand {{ cuda11_libcurand_host_version }}
+    - libcurand-dev {{ cuda11_libcurand_host_version }}
+    - libcusolver {{ cuda11_libcusolver_host_version }}
+    - libcusolver-dev {{ cuda11_libcusolver_host_version }}
+    - libcusparse {{ cuda11_libcusparse_host_version }}
+    - libcusparse-dev {{ cuda11_libcusparse_host_version }}
+    {% else %}
+    - cuda-python ==12.0.0
+    - libcublas-dev
+    - libcurand-dev
+    - libcusolver-dev
+    - libcusparse-dev
+    {% endif %}


Do we actually need the -dev packages for cuml? Shouldn't we only require the runtime packages? The dependency on -dev packages should be isolated to libcuml I believe.

I vaguely recall that the -dev packages are needed from when I looked at this months ago. I think cuml might do something with raft that requires these libraries.

I can't remember well, so did a commit trying it to check

It looks like we were able to remove this?

conda/recipes/libcuml/meta.yaml

Co-authored-by: Vyas Ramasubramani <vyas.ramasubramani@gmail.com>

bdice

One more round of questions about math library dependencies.

conda/recipes/libcuml/meta.yaml

bdice · 2023-07-12T23:27:20Z

conda/recipes/cuml/meta.yaml

+    - libcublas {{ cuda11_libcublas_host_version }}
+    - libcurand {{ cuda11_libcurand_host_version }}
+    - libcusolver {{ cuda11_libcusolver_host_version }}
+    - libcusparse {{ cuda11_libcusparse_host_version }}


Do any of these CUDA math libraries need to be in host for cuml Python? Are they directly used by cuml Python or only indirectly through libcuml? If the latter, maybe we can get away with only specifying them in libcuml's recipe.

I wonder if these (non-dev) libraries are also remnants of the changes used when migrating to GitHub Actions before RAFT recipes were complete. It would surprise me if these are used directly in cuml Python, since they're not mentioned in python/CMakeLists.txt. However, I see that there are linkages in the installed packages: for example, ldd manifold/umap.cpytho-310-x86_64-linux-gnu.so shows libcufft.so.10, libcurand.so.10, libcusolver.so.11, libcublas.so.11, libcusparse.so.11.

After thinking about this more, we probably want to remove these. Putting “runtime” (non-dev) libraries in the host section probably does nothing, except perhaps satisfy conda linking checks (which we can verify by looking at the warnings).

bdice · 2023-07-12T23:27:58Z

conda/recipes/cuml/meta.yaml

+    {% if cuda_major == "11" %}
+    - cuda-python ==11.7.1
+    - cudatoolkit
+    - libcublas {{ cuda11_libcublas_host_version }}


Is libcufft missing? If so we'd need to add it to conda_build_config.yaml for cuml, too. (See also my other comment about whether any of these libraries are needed in the Python packaging.)

Suggested change

- libcublas {{ cuda11_libcublas_host_version }}

- libcublas {{ cuda11_libcublas_host_version }}

- libcufft {{ cuda11_libcufft_host_version }}

bdice · 2023-07-12T23:31:47Z

conda/recipes/cuml/meta.yaml

+    - libcusparse {{ cuda11_libcusparse_host_version }}
+    {% else %}
+    - cuda-python ==12.0.0
+    - libcublas


Is libcufft missing? (Or can all these math libraries be removed from the Python packaging, see other comment?)

Suggested change

- libcublas

- libcublas

- libcufft

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

bdice

Thanks for the hard work on this @dantegd! LGTM.

vyasr

Everything looks good for me too (can't approve since it's my PR).

dantegd · 2023-07-13T16:52:54Z

/merge

jakirkham · 2023-07-13T17:07:31Z

Thanks everyone! 🙏

vyasr added feature request New feature or request 2 - In Progress Currenty a work in progress non-breaking Non-breaking change labels Mar 31, 2023

vyasr self-assigned this Mar 31, 2023

github-actions bot added ci conda conda issue labels Mar 31, 2023

vyasr added the 5 - DO NOT MERGE Hold off on merging; see PR for details label Mar 31, 2023

vyasr changed the title ~~Build CUDA 12 packages of libraft~~ Build CUDA 12 packages of libcuml Mar 31, 2023

vyasr added 11 commits April 6, 2023 14:33

Switch to using CUDA 12 workflows

b20cc08

Add nvcc flags needed to work with broken nvidia packages

0f9aa61

Add a cuda11 compiler version and branch out the recipes

b77fd91

Force build and test scripts to install rmm from the PR.

a42fb8e

Comment out unnecessary builds

436a1f6

Bifurcate all cuda deps by CUDA version

1b2ac1c

Pull raft and culmprims from CUDA 12 PRs

42c530e

Patch conda channels

fb2f30e

Update dependencies.yaml

9cf22b8

Remove faiss dep for CUDA 12

6954977

Update to latest raft and cumlprims hashes

0a44511

vyasr force-pushed the feat/cuda12_nvidia_build branch from ba8f9b0 to 0a44511 Compare April 6, 2023 21:41

jakirkham mentioned this pull request Apr 7, 2023

cuML: CUDA 12 Conda Packages #5234

Closed

vyasr changed the title ~~Build CUDA 12 packages of libcuml~~ libcuml: Build CUDA 12 packages Apr 10, 2023

vyasr changed the title ~~libcuml: Build CUDA 12 packages~~ cuml: Build CUDA 12 packages Apr 10, 2023

dantegd added 6 commits April 12, 2023 21:30

UPD Update RMM hash

347195b

Empty commit to get CI fix

424ee3c

Merge branch-23.06

188c8da

FIX line from bad merge

d5fbb32

DBG remove conda-forge channel packages before installing artifacts

422b49a

FIX Remove cudf from libcuml conda recipe

c51a1d5

FIX temporarily skip test_sparse_pca_inputs in CUDA 12

85960da

viclafargue and others added 2 commits July 11, 2023 14:16

Fix PCA test

f40b7f9

FIX Merge fix of PR rapidsai#5498

1d06c86

dantegd marked this pull request as ready for review July 11, 2023 13:50

dantegd requested review from a team as code owners July 11, 2023 13:50

FIX Remove not used import

325e954

bdice added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currenty a work in progress labels Jul 11, 2023

bdice reviewed Jul 11, 2023

View reviewed changes

dantegd and others added 3 commits July 11, 2023 18:12

Apply suggestions from code review

b3cee15

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

FIX remove dev packages from run reqs of libcuml

e107588

FIX forgot to run rdfg

b186129

vyasr commented Jul 11, 2023

View reviewed changes

dantegd and others added 2 commits July 12, 2023 14:41

Update conda/recipes/libcuml/meta.yaml

94995c1

Co-authored-by: Vyas Ramasubramani <vyas.ramasubramani@gmail.com>

DBG remove dev ctk packages from cuml host

a301e01

bdice reviewed Jul 12, 2023

View reviewed changes

dantegd and others added 2 commits July 12, 2023 22:56

Apply suggestions from code review

fe27991

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

DBG remove cuda libraries from cuml host in recipe

7ce78b4

bdice approved these changes Jul 13, 2023

View reviewed changes

raydouglass approved these changes Jul 13, 2023

View reviewed changes

vyasr commented Jul 13, 2023

View reviewed changes

wphicks approved these changes Jul 13, 2023

View reviewed changes

jakirkham approved these changes Jul 13, 2023

View reviewed changes

rapids-bot bot merged commit c3d4dbb into rapidsai:branch-23.08 Jul 13, 2023

pxLi mentioned this pull request Aug 9, 2023

Upgrade cuml from 23.06 to 23.08 in pytests and ci NVIDIA/spark-rapids-ml#353

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cuml: Build CUDA 12 packages #5318

cuml: Build CUDA 12 packages #5318

vyasr commented Mar 31, 2023 •

edited by bdice

Loading

vyasr commented Mar 31, 2023

dantegd commented Jul 11, 2023

dantegd commented Jul 11, 2023

bdice left a comment

vyasr left a comment

vyasr Jul 11, 2023

bdice Jul 12, 2023

dantegd Jul 12, 2023

vyasr Jul 13, 2023

bdice left a comment

bdice Jul 12, 2023

bdice Jul 13, 2023

bdice Jul 12, 2023 •

edited

Loading

bdice Jul 12, 2023 •

edited

Loading

bdice left a comment

vyasr left a comment

dantegd commented Jul 13, 2023

jakirkham commented Jul 13, 2023

	- libcublas {{ cuda11_libcublas_host_version }}
	- libcublas {{ cuda11_libcublas_host_version }}
	- libcufft {{ cuda11_libcufft_host_version }}

cuml: Build CUDA 12 packages #5318

cuml: Build CUDA 12 packages #5318

Conversation

vyasr commented Mar 31, 2023 • edited by bdice Loading

vyasr commented Mar 31, 2023

dantegd commented Jul 11, 2023

dantegd commented Jul 11, 2023

bdice left a comment

Choose a reason for hiding this comment

vyasr left a comment

Choose a reason for hiding this comment

vyasr Jul 11, 2023

Choose a reason for hiding this comment

bdice Jul 12, 2023

Choose a reason for hiding this comment

dantegd Jul 12, 2023

Choose a reason for hiding this comment

vyasr Jul 13, 2023

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

bdice Jul 12, 2023

Choose a reason for hiding this comment

bdice Jul 13, 2023

Choose a reason for hiding this comment

bdice Jul 12, 2023 • edited Loading

Choose a reason for hiding this comment

bdice Jul 12, 2023 • edited Loading

Choose a reason for hiding this comment

bdice left a comment

Choose a reason for hiding this comment

vyasr left a comment

Choose a reason for hiding this comment

dantegd commented Jul 13, 2023

jakirkham commented Jul 13, 2023

vyasr commented Mar 31, 2023 •

edited by bdice

Loading

bdice Jul 12, 2023 •

edited

Loading

bdice Jul 12, 2023 •

edited

Loading