Add support for Python 3.11 #3

bdice · 2023-12-13T17:41:55Z

RAPIDS currently supports Python 3.9 and 3.10. We would like to add support for Python 3.11. This issue documents the steps needed.

Each section should be fully completed before moving to the next section.

CI images

Add Python 3.11 builds of miniforge-cuda. Drop CUDA 11.2 and 11.5, add Python 3.11 and 3.12. miniforge-cuda#55
Add Python 3.11 CI images (ci-conda, ci-wheel, citestwheel) Drop CUDA 11.2 and 11.5, add Python 3.11 ci-imgs#96

CI workflows

Branch Strategy:

Create a branch on shared-workflows called python-3.11
Add Python 3.11 to the build matrix on the python-3.11 branch: https://github.com/rapidsai/shared-workflows/blob/e7ebbae5854727b897b65213cf51ff8b965f53c1/.github/workflows/conda-python-build.yaml#L56-L63
Add Python 3.11 to the test matrix on the python-3.11 branch: https://github.com/rapidsai/shared-workflows/blob/e7ebbae5854727b897b65213cf51ff8b965f53c1/.github/workflows/conda-python-tests.yaml#L68-L82
- When adjusting the test matrix, be aware of total GPU resource consumption. Build jobs are CPU only but test jobs require GPUs. We want to keep our GPU consumption roughly the same (don't double the test matrix size), even if it gets a bit sparser in its coverage. I would recommend changing a portion of the Python 3.10 jobs to use Python 3.11. Try to mix coverage across architectures, OS versions, Python versions, and CUDA versions, and be sure we cover configurations like "latest of everything (OS/Python/CUDA)" and "oldest of everything."

Experimental Strategy:

@ajschmidt8 proposed that we could use GitHub Actions' continue-on-error to add Python 3.11 jobs that are allowed to fail, for all repos. Once all repos are passing, we could require the job to pass. This could be cleaner and less total work than our normal approach of the branch strategy above.

RAPIDS repositories

I copied a list of repositories from https://github.com/rapidsai/workflows/blob/dfd73ad47d977c57ee27b2349216f85946be757b/.github/workflows/nightly-pipeline.yaml#L27-L44 and sorted it roughly in dependency-tree order.

For each repo,

~~Update .github/workflows/*.yaml to point to the python-3.11 branch of shared-workflows~~
Update dependencies.yaml to add support for Python 3.11.
Review any pyproject.toml files for necessary changes (classifiers, etc.)
Update docs (README, etc) that reference a single Python version to point to the latest (3.11).
Once CI passes, merge the PR.

Most of this is easy to automate with rapids-reviser, and I've made an attempt at it here: https://github.com/rapidsai/rapids-reviser/pull/11. We still need to manually review the PRs for missing pieces.

Repos:

(checklist moved to #3 (comment))

Once all repos are migrated to the python-3.11 branch, the migration is complete. We merge python-3.11 into branch-24.02 on shared-workflows and then open follow-up PRs to each repo to reset the branches to branch-24.02. This "reset" is simple and should be automated with rapids-reviser.

Post-migration

Update the ci-imgs repo's latest configuration to use Python 3.11. https://github.com/rapidsai/ci-imgs/blob/main/latest.yaml
- The latest image is frequently used by CI jobs for building docs and testing notebooks. Be aware that issues may arise in those jobs.
- PR: Use Python 3.11 in latest tag. ci-imgs#125
Update the docker repo's matrix.yaml and matrix-test.yaml
- PR: Add Python 3.11. docker#635
(added by @jameslamb 2/27) update the build matrix in pypi-wheel-scripts so that Python 3.11 wheels are checked: https://github.com/rapidsai/pypi-wheel-scripts/blob/fa1e8744c8ec961a5b5e38ae172ae9c8c51b4280/release/check-wheels.sh#L41-L45
- PR: https://github.com/rapidsai/pypi-wheel-scripts/pull/8

The text was updated successfully, but these errors were encountered:

jameslamb · 2024-02-26T15:43:50Z

Contributes to rapidsai/build-planning#3 This PR adds support for Python 3.11. ## Notes for Reviewers This is part of ongoing work to add Python 3.11 support across RAPIDS. The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged. This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11. See rapidsai/shared-workflows#176 for more details. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Kyle Edwards (https://github.com/KyleFromNVIDIA) URL: #547

Contributes to rapidsai/build-planning#3 This PR adds support for Python 3.11. ## Notes for Reviewers This is part of ongoing work to add Python 3.11 support across RAPIDS. The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged. This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11. See rapidsai/shared-workflows#176 for more details. Authors: - James Lamb (https://github.com/jameslamb) - Bradley Dice (https://github.com/bdice) Approvers: - Bradley Dice (https://github.com/bdice) - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Jake Awe (https://github.com/AyodeAwe) URL: #1469

KyleFromNVIDIA · 2024-02-27T15:19:51Z

Updated project order to reflect the fact that dask-cuda has a dependency on cudf.

jameslamb · 2024-02-27T16:46:08Z

Added an item to the description about updating this set of checks in pypi-wheel-scripts: https://github.com/rapidsai/pypi-wheel-scripts/blob/fa1e8744c8ec961a5b5e38ae172ae9c8c51b4280/release/check-wheels.sh#L41-L45

bdice · 2024-02-27T16:49:51Z

Updated project order to reflect the fact that dask-cuda has a dependency on cudf.

Unfortunately there's a circular dependency between these repos in their test dependencies. We typically have to admin-merge the dask-cuda PR whenever new packages are created.

dask-cuda requires dask-cudf: https://github.com/rapidsai/dask-cuda/blob/5a0a536c80594873a2907dca69a4ae58b5623d7d/dependencies.yaml#L161-L162

dask-cudf requires dask-cuda: https://github.com/rapidsai/cudf/blob/ef20ef9af942d840effd8ac16f8b302a8e4b1dd4/dependencies.yaml#L646-L650

Contributes to rapidsai/build-planning#3 This PR adds support for Python 3.11. ## Notes for Reviewers This is part of ongoing work to add Python 3.11 support across RAPIDS. The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged. This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11. See rapidsai/shared-workflows#176 for more details. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Jake Awe (https://github.com/AyodeAwe) URL: #1315

Contributes to rapidsai/build-planning#3 This PR adds support for Python 3.11. It also bumps uses of `NumPy` to `numpy>=1.23`, see rapidsai/build-planning#3 (comment). ## Notes for Reviewers This is part of ongoing work to add Python 3.11 support across RAPIDS. The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged. This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11. See rapidsai/shared-workflows#176 for more details. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Jake Awe (https://github.com/AyodeAwe) - Bradley Dice (https://github.com/bdice) URL: #139

Contributes to rapidsai/build-planning#3 This PR adds support for Python 3.11. It also bumps uses of `NumPy` to `numpy>=1.23`, see rapidsai/build-planning#3 (comment). ## Notes for Reviewers This is part of ongoing work to add Python 3.11 support across RAPIDS. The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged. This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11. See rapidsai/shared-workflows#176 for more details. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Jake Awe (https://github.com/AyodeAwe) - https://github.com/jakirkham - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Bradley Dice (https://github.com/bdice) - Corey J. Nolet (https://github.com/cjnolet) URL: #2200

Contributes to rapidsai/build-planning#3 This PR adds support for Python 3.11. It also bumps uses of `NumPy` to `numpy>=1.23`, see rapidsai/build-planning#3 (comment). ## Notes for Reviewers This is part of ongoing work to add Python 3.11 support across RAPIDS. The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged. This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11. See rapidsai/shared-workflows#176 for more details. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Jake Awe (https://github.com/AyodeAwe) - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Rick Ratzel (https://github.com/rlratzel) URL: #4195

Contributes to rapidsai/build-planning#3 This PR adds support for Python 3.11. It also bumps uses of `NumPy` to `numpy>=1.23`, see rapidsai/build-planning#3 (comment). ## Notes for Reviewers This is part of ongoing work to add Python 3.11 support across RAPIDS. The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged. This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11. See rapidsai/shared-workflows#176 for more details. *(created with [rapids-reviser](https://github.com/rapidsai/rapids-reviser))* Authors: - James Lamb (https://github.com/jameslamb) - Kyle Edwards (https://github.com/KyleFromNVIDIA) - https://github.com/jakirkham - Bradley Dice (https://github.com/bdice) - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Bradley Dice (https://github.com/bdice) - Kyle Edwards (https://github.com/KyleFromNVIDIA) - Jake Awe (https://github.com/AyodeAwe) - GALI PREM SAGAR (https://github.com/galipremsagar) URL: #15111

This marks a memory test as `xfail`, which was failing on Python 3.11. xref: #1352 xref: rapidsai/build-planning#3 Authors: - Kyle Edwards (https://github.com/KyleFromNVIDIA) Approvers: - Bradley Dice (https://github.com/bdice) - https://github.com/jakirkham - Vyas Ramasubramani (https://github.com/vyasr) URL: #1351

bdice · 2024-03-01T22:49:35Z

We are currently working on adding cubinlinker and ptxcompiler builds for Python 3.11. These are manual processes, so they are hard to track with PRs.

cc: @KyleFromNVIDIA @vyasr @gmarkall

cubinlinker wheels for Python 3.11 on nightly PyPI repo
cubinlinker wheels for Python 3.11 on NVIDIA PyPI repo
ptxcompiler wheels for Python 3.11 on nightly PyPI repo
ptxcompiler wheels for Python 3.11 on NVIDIA PyPI repo
cubinlinker conda packages for Python 3.11 on rapidsai / rapidsai-nightly channels
(ptxcompiler conda packages are handled by https://github.com/conda-forge/ptxcompiler-feedstock/ and already exist)

vyasr · 2024-03-01T23:23:18Z

Wheels for cubinlinker and ptxcompiler are now available on our Anaconda PyPI repo (see e.g. https://anaconda.org/rapidsai-wheels-nightly/ptxcompiler-cu11/files).

jakirkham · 2024-03-02T00:49:22Z

cubinlinker conda packages for Python 3.11 on rapidsai / rapidsai-nightly channels

Looking at the cubinlinker packages on the channels:

It looks like Python 3.11 packages are already available

bdice · 2024-03-04T14:36:30Z

There was an additional step needed to copy cubinlinker 0.3.0 build 1 from rapidsai-nightly to rapidsai. It has now been completed. https://anaconda.org/rapidsai/cubinlinker/files?version=0.3.0

The last items are getting cubinlinker/ptxcompiler wheels copied from the nightly PyPI to the NVIDIA PyPI repo.

jakirkham · 2024-03-04T19:42:30Z

For clarity that was done for Python 3.12 to future proof that upgrade

Adds support for Python 3.11. xref: rapidsai/build-planning#3 Authors: - Bradley Dice (https://github.com/bdice) Approvers: - Ray Douglass (https://github.com/raydouglass) URL: #635

jakirkham · 2024-03-05T18:39:31Z

Closing as completed

bdice · 2024-03-07T18:55:41Z

We missed a step. I added Python 3.11 to the nightly release selector here: rapidsai/docs#496

jakirkham · 2024-03-07T19:15:25Z

Good catch. Thanks Bradley! 🙏

Contributes to rapidsai/build-planning#3 This PR adds support for Python 3.11. ## Notes for Reviewers This is part of ongoing work to add Python 3.11 support across RAPIDS. The Python 3.11 CI workflows introduced in rapidsai/shared-workflows#176 are *optional*... they are not yet required to run successfully for PRs to be merged. This PR can be merged once all jobs are running successfully (including the non-required jobs for Python 3.11). The CI logs should be verified that the jobs are building and testing with Python 3.11. See rapidsai/shared-workflows#176 for more details. Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Bradley Dice (https://github.com/bdice) - Jake Awe (https://github.com/AyodeAwe) URL: rapidsai#1315

As NumPy 1.23 is needed for Python 3.11 support, go ahead and bump the minimum NumPy version used by Dask-CUDA to match that. xref: rapidsai#1315 xref: rapidsai/build-planning#3 Authors: - https://github.com/jakirkham - Bradley Dice (https://github.com/bdice) Approvers: - Mads R. B. Kristensen (https://github.com/madsbk) - Peter Andreas Entschev (https://github.com/pentschev) - Jake Awe (https://github.com/AyodeAwe) URL: rapidsai#1316

jameslamb changed the title ~~Add support for Python 3~~ Add support for Python 3.11 Dec 14, 2023

bdice mentioned this issue Jan 6, 2024

run wheels-publish jobs in Python 3.11 image rapidsai/shared-workflows#169

Merged

bdice mentioned this issue Feb 1, 2024

[FEA] Python3.11 support rapidsai/cudf#13314

Closed

jameslamb assigned msarahan and jameslamb and unassigned msarahan Feb 20, 2024

This was referenced Feb 20, 2024

add support for python 3.11 rapidsai/shared-workflows#176

Merged

Add support for Python 3.11 rapidsai/rmm#1469

Merged

re-organize build matrices rapidsai/shared-workflows#177

Merged

Add support for Python 3.11, require NumPy 1.23+ rapidsai/cudf#15111

Merged

jakirkham mentioned this issue Feb 26, 2024

Evaluate using scikit-build-core rapidsai/cucim#645

Open

This was referenced Feb 27, 2024

Add support for Python 3.11 rapidsai/ucx-py#1027

Merged

Add support for Python 3.11, require NumPy 1.23+ rapidsai/ucxx#199

Merged

jakirkham mentioned this issue Feb 29, 2024

Fix Python 3.11 test failures rapidsai/cuspatial#1351

Merged

3 tasks

jakirkham mentioned this issue Mar 1, 2024

[NO MRG] Test Python 3.11 on CI rapidsai/integration#702

Closed

jameslamb mentioned this issue Mar 1, 2024

make Python 3.11 jobs required rapidsai/shared-workflows#185

Merged

This was referenced Mar 1, 2024

Use Python 3.11 in latest tag. rapidsai/ci-imgs#125

Merged

Add Python 3.11. rapidsai/docker#635

Merged

KyleFromNVIDIA mentioned this issue Mar 1, 2024

Fix wheel build failures with Python 3.11 rapidsai/cuspatial#1353

Closed

3 tasks

jakirkham closed this as completed Mar 5, 2024

jakirkham mentioned this issue Mar 14, 2024

Unpin numpy in Dask environment, leave dask pinned in Dask-SQL environment rapidsai/dask-build-environment#88

Merged

This was referenced Mar 19, 2024

Start building images for python 3.11 rapidsai/dask-build-environment#91

Merged

Add python 3.11 to GPU CI matrix dask/distributed#8598

Merged

VibhuJawa mentioned this issue Mar 27, 2024

Bump Python and RAPIDS versions NVIDIA/NeMo-Curator#16

Merged

bdice mentioned this issue Apr 18, 2024

Add support for Python 3.12 #40

Closed

5 tasks

jakirkham mentioned this issue May 15, 2024

Updating Python versions rapidsai/dask-build-environment#93

Closed

jakirkham added the epic label May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Python 3.11 #3

Add support for Python 3.11 #3

bdice commented Dec 13, 2023 •

edited by jakirkham

Loading

jameslamb commented Feb 26, 2024 •

edited by KyleFromNVIDIA

Loading

KyleFromNVIDIA commented Feb 27, 2024

jameslamb commented Feb 27, 2024

bdice commented Feb 27, 2024

bdice commented Mar 1, 2024 •

edited by jakirkham

Loading

vyasr commented Mar 1, 2024

jakirkham commented Mar 2, 2024

bdice commented Mar 4, 2024

jakirkham commented Mar 4, 2024

jakirkham commented Mar 5, 2024

bdice commented Mar 7, 2024

jakirkham commented Mar 7, 2024

Add support for Python 3.11 #3

Add support for Python 3.11 #3

Comments

bdice commented Dec 13, 2023 • edited by jakirkham Loading

CI images

CI workflows

RAPIDS repositories

Post-migration

jameslamb commented Feb 26, 2024 • edited by KyleFromNVIDIA Loading

KyleFromNVIDIA commented Feb 27, 2024

jameslamb commented Feb 27, 2024

bdice commented Feb 27, 2024

bdice commented Mar 1, 2024 • edited by jakirkham Loading

vyasr commented Mar 1, 2024

jakirkham commented Mar 2, 2024

bdice commented Mar 4, 2024

jakirkham commented Mar 4, 2024

jakirkham commented Mar 5, 2024

bdice commented Mar 7, 2024

jakirkham commented Mar 7, 2024

bdice commented Dec 13, 2023 •

edited by jakirkham

Loading

jameslamb commented Feb 26, 2024 •

edited by KyleFromNVIDIA

Loading

bdice commented Mar 1, 2024 •

edited by jakirkham

Loading