run package uploads in python slim containers by jameslamb · Pull Request #586 · rapidsai/shared-workflows

jameslamb · 2026-06-29T18:52:37Z

Contributes to #505

As described there, it's not uncommon for pulling the rapidsai/ci-wheel images to take 3-4 minutes.

This is extra painful for jobs like wheels-publish, which just need lightweight publishing tools and not CUDA libraries, compiler toolchain, etc.

This PR proposes working around that by doing the following:

using a small python:3.14-slim image instead
installing just the small set of necessary tools at runtime of the wheels-publish job

Notes for Reviewers

Benefits

faster conda-uploads-packages and wheels-publish, which means reduced end-to-end time for RAPIDS nightly pipeline (see "how I tested this")
would allow us to stop installing anaconda-client and its dependencies in the rapidsai/ci-conda and rapidsai/ci-wheel images at https://github.com/rapidsai/ci-imgs
- that project has a lot of dependencies, so that might have a small but notable improvement in image size, build time, and pull time
- that'd affect all wheel builds and conda building + testing jobs

Costs / Risks

adds more network calls and package installs at runtime of wheels-publish, which might lead to more transient failures from network issues
adds complexity to the workflow code

Why not pre-build a `wheels-publish` image?

That WOULD make this even faster and avoid all those package installs at runtime.

But new images need to go through a compliance/legal approval process that's a bit heavier than this use case justifies, in my opinion. In my testing, installing tools took 15-20 seconds, so that's the most we'd save by having a pre-built image with everything installed.

How I tested this

rmm's main branch was already pointed at this branch for package-uploading jobs from #585.

Put up changes in a gha-tools branch (rapidsai/gha-tools#265) to force-overwrite existing packages, so the upload time to anaconda.org is included in the timings.

Clicked "re-run all jobs" on rmm's most recent run on main.

workflow	previous `main` (as of #585)	this PR
upload-conda	1m32s	1m15s
wheels-publish-cpp	2m51s	0m37s
wheels-publish-python	3m3s	0m41s

builds from rmm: (previous main | this PR)

jameslamb · 2026-06-29T19:35:54Z

      # CUDA toolkit version of the container is irrelevant in the publish step.
      # This just uploads already-built wheels to remote storage.
-      image: "rapidsai/ci-wheel:26.08-latest" # zizmor: ignore[unpinned-images]
+      image: "python:3.14-slim"


We shouldn't meaningfully be impacted by Python version here, as long as it's an officially-supported version.

I chose not to use python:latest here to reduce the risk of these builds failing if :latest becomes (for example) 3.16.0rc1 and there aren't yet compatible wheels from some dependencies we pip install.

jameslamb · 2026-06-29T19:36:25Z

+      uses: nv-gha-runners/setup-proxy-cache@main
+      continue-on-error: true
+      with:
+        enable-apt: true


Now that code in this workflow runs apt-get install and pip install, we'll want the proxy cache for faster downloads and reduced risk of network issues.

jameslamb · 2026-06-29T19:37:18Z

+        # install gha-tools
+        git clone --depth 1 --branch test-uploads https://github.com/rapidsai/gha-tools /tmp/gha-tools
+        echo "/tmp/gha-tools/tools" >> "${GITHUB_PATH}"
+        export PATH="/tmp/gha-tools/tools:${PATH}"


GITHUB_PATH doesn't take effect until the next step, but we want gha-tools to be on PATH for rapids-pip-retry in this step.

jameslamb · 2026-06-29T19:37:53Z

-        else
-            gh api /rate_limit | jq .
-        fi
+        gh api /rate_limit | jq .


We're now explicitly installing gh as part of this workflow. gh not being available would be a bug and we should get a loud error telling us about it.

gforsyth · 2026-06-29T20:05:41Z

I agree that ci-wheel is comically oversized and that we should fix that.

Question: with the CuPy floor getting bumped up to >=14, should we do this and switch over to using cuda-base from cuda-devel?

I believe the cupy floor was the last thing (pending some tests runs...) holding us back from switching over to smaller images

bdice

Looks fine to me. Feel free to use your admin powers to test with RMM as needed.

bdice · 2026-06-29T20:26:56Z

I agree that ci-wheel is comically oversized and that we should fix that.

Question: with the CuPy floor getting bumped up to >=14, should we do this and switch over to using cuda-base from cuda-devel?

I believe the cupy floor was the last thing (pending some tests runs...) holding us back from switching over to smaller images

Yes, I think that's correct -- let's try to revive rapidsai/ci-imgs#408 once rapidsai/build-planning#279 is complete.

jameslamb · 2026-06-29T20:44:19Z

Question: with the CuPy floor getting bumped up to >=14, should we do this and switch over to using cuda-base from cuda-devel?

I think yes, we should do BOTH, and we shouldn't couple them.

Using a smaller base image for ci-wheel would make the gain here smaller, but some of it is independent of which nvidai/cuda image is the base... it's still true that we don't need any CUDA libraries to publish wheels or conda packages.

And this PR has the benefit of totally decoupling publishing from the content of the ci-imgs images, which opens up more savings like removing the anaconda-client package from those images (a space saving regardless of base image).

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

Co-authored-by: James Lamb <jaylamb20@gmail.com>

jameslamb · 2026-06-29T20:54:42Z

/merge

## Description Follow-up to #2458 Testing in `shared-workflows` is complete now that rapidsai/shared-workflows#586 has been merged. This resets all branch references for `shared-workflows` back to `main`. ## Checklist - [x] I am familiar with the [Contributing Guidelines](https://github.com/rapidsai/rmm/blob/HEAD/CONTRIBUTING.md). - [x] New or existing tests cover these changes. - [x] The documentation is up to date with these changes.

jameslamb · 2026-06-29T20:57:24Z

I've reset rmm's references to @main: rapidsai/rmm#2459

rapidsai/shared-workflows#586 switched the `conda-upload-packages` and `wheels-publish` jobs away from using this repo's images... they new use a `python:*-slim` image and install the tools they need at runtime. As a result, `rapidsai/ci-{conda,wheel}` no longer need to install `anaconda client`, the client library for anaconda.org. This proposes removing it. It pulls in a lot of dependencies, so this should help a bit with build times and image size (don't have estimates). ## Notes for Reviewers ### Is this safe? I think so. It appears the only remaining use of this is in `rapidsai/legate-{boost,dataframe}`, which aren't being actively developed: * `rapidsai`: https://github.com/search?q=org%3Arapidsai+%22anaconda+%22+AND+NOT+is%3Aarchived&type=code * `NVIDIA`: https://github.com/search?q=org%3ANVIDIA+%22anaconda+%22+AND+NOT+is%3Aarchived&type=code&p=3 Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Gil Forsyth (https://github.com/gforsyth) URL: #432

run package uploads in python slim containers

2f33ba1

jameslamb added improvement Improves an existing functionality non-breaking Introduces a non-breaking change labels Jun 29, 2026

jameslamb mentioned this pull request Jun 29, 2026

[NEVER MERGE] remove --skip-existing from upload tools rapidsai/gha-tools#265

Closed

simplify tool installs, fix cloning

8b390aa

jameslamb commented Jun 29, 2026

View reviewed changes

Comment thread .github/workflows/wheels-publish.yaml Outdated

jameslamb commented Jun 29, 2026

View reviewed changes

Comment thread .github/workflows/conda-upload-packages.yaml Outdated

jameslamb changed the title ~~WIP: run package uploads in python slim containers~~ run package uploads in python slim containers Jun 29, 2026

jameslamb requested review from bdice and gforsyth June 29, 2026 19:41

jameslamb marked this pull request as ready for review June 29, 2026 19:41

jameslamb requested a review from a team as a code owner June 29, 2026 19:41

bdice approved these changes Jun 29, 2026

View reviewed changes

Comment thread .github/workflows/conda-upload-packages.yaml Outdated

jameslamb and others added 3 commits June 29, 2026 15:44

Update .github/workflows/conda-upload-packages.yaml

1aca284

Co-authored-by: Bradley Dice <bdice@bradleydice.com>

use 'main' of gha-tools

0acff9b

Co-authored-by: James Lamb <jaylamb20@gmail.com>

oh Github browser editing, why are you so weird about whitespace

c395258

jameslamb mentioned this pull request Jun 29, 2026

wheels CI: reduce CI time #505

Open

7 tasks

rapids-bot Bot merged commit aed0820 into main Jun 29, 2026
2 checks passed

jameslamb deleted the cheaper-checkout branch June 29, 2026 20:55

jameslamb mentioned this pull request Jun 29, 2026

swap shared-workflows branch back to 'main' rapidsai/rmm#2459

Merged

3 tasks

jameslamb mentioned this pull request Jun 29, 2026

remove 'anaconda-client' rapidsai/ci-imgs#432

Merged

Navigation Menu

Uh oh!

run package uploads in python slim containers - #586

run package uploads in python slim containers#586
rapids-bot[bot] merged 5 commits into
mainfrom
cheaper-checkout

jameslamb commented Jun 29, 2026 •

edited

Loading

Uh oh!

jameslamb Jun 29, 2026

Uh oh!

jameslamb Jun 29, 2026

Uh oh!

Uh oh!

jameslamb Jun 29, 2026

Uh oh!

jameslamb Jun 29, 2026

Uh oh!

Uh oh!

gforsyth commented Jun 29, 2026 •

edited

Loading

Uh oh!

bdice left a comment

Uh oh!

Uh oh!

bdice commented Jun 29, 2026

Uh oh!

jameslamb commented Jun 29, 2026

Uh oh!

jameslamb commented Jun 29, 2026

Uh oh!

Uh oh!

jameslamb commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jameslamb commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes for Reviewers

Benefits

Costs / Risks

Why not pre-build a wheels-publish image?

How I tested this

Uh oh!

jameslamb Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

jameslamb Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jameslamb Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

jameslamb Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gforsyth commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bdice left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bdice commented Jun 29, 2026

Uh oh!

jameslamb commented Jun 29, 2026

Uh oh!

jameslamb commented Jun 29, 2026

Uh oh!

Uh oh!

jameslamb commented Jun 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jameslamb commented Jun 29, 2026 •

edited

Loading

Why not pre-build a `wheels-publish` image?

gforsyth commented Jun 29, 2026 •

edited

Loading