[ci] Nightly for building+uploading manylinux baseimage by andrew-anyscale · Pull Request #59204 · ray-project/ray

andrew-anyscale · 2025-12-05T20:28:58Z

Adds new buildkite job for building + uploading a base manylinux2014 image. Planning to build in flavors of with and without JDK included to help reduce image size when JDK is not needed.

Building is handled by a wanda builder step
Uploading to rayproject/manylinux2014:{DATE}.{SHORT_COMMIT}-x86_64 and rayproject/manylinux2014:{DATE}.{SHORT_COMMIT}-jdk-x86_64 is handled by a separate runner that calls the new ci/ray_ci/automation/copy_wanda_image.py`
Runs by default without uploading for initial testing. Once passed with flag --upload, it will upload to the specified registry

gemini-code-assist

Code Review

This pull request introduces a set of new shell scripts to automate the nightly build and upload of the manylinux base image. The code is well-organized into a main script and reusable utility libraries. The use of shell best practices like set -euo pipefail and proper variable quoting is commendable. My review provides a few suggestions to further improve the robustness and efficiency of the utility scripts, such as adding include guards, redirecting error output to stderr, and conditionally defining shared variables. Overall, this is a solid addition to the CI process.

ci/base_images/utils/common.sh

ci/base_images/utils/lib-docker.sh

ci/base_images/nightly/build-manylinux.sh

.buildkite/cicd-cron/cicd-cron.rayci.yml

ci/base_images/nightly/push-manylinux.sh

andrew-anyscale · 2025-12-09T15:46:18Z

.buildkite/cicd-cron/cicd-cron.rayci.yml

+      RAYCI_DISABLE_JAVA: "true"
+      JDK_SUFFIX: ""


I wanted to use a {{matrix}} operation here, but couldn't find a way to set RAYCI_DISABLE_JAVA to true and cover the JDK_SUFFIX for the image name...

I think this is fine. the matrix only has two elements, so not a big deal..

we probably should also have the manylinux aarch64 ones here. for completeness.

andrew-anyscale · 2025-12-09T16:12:56Z

.buildkite/cicd-cron/cicd-cron.rayci.yml

+steps:
+  - name: manylinux-nightly-no-jdk
+    label: ":crescent_moon: wanda: manylinux-nightly (no JDK)"
+    wanda: ci/docker/manylinux-nightly.wanda.yaml


Is there a way to pass -rebuild via this interface? https://github.com/ray-project/rayci/blob/main/wanda/wanda/main.go#L42

I also found DisableCaching, which I believe would require a new wanda.yaml definition-- https://github.com/ray-project/rayci/blob/main/wanda/spec.go#L30

I looked into the code, and it does not seem to have one.

I think we can add a global env var like RAYCI_WANDA_ALWAYS_REBUILD to control the behavior.

also, I realized that force-rebuild is that critical. the main purpose is really to have a pre-built one, and rayci+wanda will by default invalidate the cache in around 3 or 4 days.

Adding in ray-project/rayci#322

ci/base_images/nightly/push-manylinux.sh

.buildkite/cicd-cron/cicd-cron.rayci.yml

aslonnie · 2025-12-10T00:35:40Z

ci/docker/manylinux-nightly.wanda.yaml

@@ -0,0 +1,10 @@
+name: "manylinux-nightly$JDK_SUFFIX"


since this is a SUFFIX, can we just reuse the existing wanda file rather than creating a new one?

If I use the same existing wanda file with the same name, how do I make sure that the JDK and non-JDK versions are passed from the respective Builder -> Uploader?

I.e. today it looks soemthing like:

WANDA_IMAGE_NAME="manylinux-nightly${JDK_SUFFIX}" WANDA_TAG="${RAYCI_WORK_REPO}:${RAYCI_BUILD_ID}-${WANDA_IMAGE_NAME}" # TODO: Change to `crane` docker pull "$WANDA_TAG"

Which depends on the ${JDK_SUFFIX} for differentiating the two images. What can I do instead for getting the correct image created+uploaded by the Builder?

aslonnie · 2025-12-10T00:37:54Z

ci/base_images/nightly/push-manylinux.sh

+
+printHeader "Pulling image from Wanda cache"
+printInfo "Pulling: ${WANDA_TAG}"
+docker pull "$WANDA_TAG"


can we use crane cp? so that we do not need to pull down the common layers.

aslonnie · 2025-12-10T00:40:30Z

ci/base_images/nightly/push-manylinux.sh

+: "${JDK_SUFFIX:?Error: JDK_SUFFIX is not set}"
+UPLOAD=${UPLOAD:-"false"}
+
+COMMIT_HASH=$(git rev-parse HEAD)


hate to ask.. but can we rewrite this script in bazel py_binary so that the logic is unit-testable and more maintainable? most ci scripts / tools live in ci/ray_ci directories.

which means you would also need to get a forge env for running the bazel py_binary I think

have you tried doing a dry run on the new cicd-cron pipeline yet?

Reasonable ask! I can rewrite as a py_binary

have you tried doing a dry run on the new cicd-cron pipeline yet?

I have not yet

It has been py_binary-d https://github.com/ray-project/ray/pull/59204/files#diff-0d54920723c7187d8f71dbd0f9b62c5f9673575ba155040b2e53a04badb8d6d2

Also refactored crane library so we could use those #59360

.buildkite/cicd-cron/cicd-cron.rayci.yml

aslonnie · 2025-12-10T00:52:16Z

.buildkite/cicd-cron/cicd-cron.rayci.yml

+steps:
+  - name: manylinux-nightly-no-jdk
+    label: ":crescent_moon: wanda: manylinux-nightly (no JDK)"
+    wanda: ci/docker/manylinux-nightly.wanda.yaml


I looked into the code, and it does not seem to have one.

I think we can add a global env var like RAYCI_WANDA_ALWAYS_REBUILD to control the behavior.

also, I realized that force-rebuild is that critical. the main purpose is really to have a pre-built one, and rayci+wanda will by default invalidate the cache in around 3 or 4 days.

aslonnie · 2025-12-10T00:55:20Z

.buildkite/cicd-cron/cicd-cron.rayci.yml

+      RAYCI_DISABLE_JAVA: "true"
+      JDK_SUFFIX: ""


I think this is fine. the matrix only has two elements, so not a big deal..

we probably should also have the manylinux aarch64 ones here. for completeness.

Signed-off-by: andrew <andrew@anyscale.com>

Now installs and uses Wanda to build a JDK and non-JDK version of manylinux. Depends on new upload flag to push to Dockerhub once that is enabled Focused heavily on error handling and being able to run from anywhere in the repo Signed-off-by: andrew <andrew@anyscale.com>

Signed-off-by: andrew <andrew@anyscale.com>

Scoped to just run build-many script. Signed-off-by: andrew <andrew@anyscale.com>

Signed-off-by: andrew <andrew@anyscale.com>

We use wanda now Signed-off-by: andrew <andrew@anyscale.com>

Signed-off-by: andrew <andrew@anyscale.com>

To be used in ray-project/ray#59204 Signed-off-by: andrew <andrew@anyscale.com>

Signed-off-by: andrew <andrew@anyscale.com>

cursor · 2025-12-15T22:02:10Z

.buildkite/cicd-cron/cicd-cron.rayci.yml

+        --destination-repository rayproject/manylinux2014
+        --tag-suffix -{{matrix}}
+
+  - label: ":docker: push: Push manylinux-cibase-jdk to Docker Hub"


Bug: Missing matrix variable in JDK push step label

The label for the JDK push step is missing {{matrix}} in the label string. Line 30 correctly uses Push manylinux-cibase-{{matrix}} to Docker Hub but line 45 uses Push manylinux-cibase-jdk to Docker Hub without the matrix variable. This causes both x86_64 and aarch64 variants of this step to have identical labels in the CI UI, which is inconsistent with the non-JDK push step pattern.

.buildkite/cicd-cron/cicd-cron.rayci.yml

Signed-off-by: andrew <andrew@anyscale.com>

cursor · 2025-12-15T22:14:31Z

.buildkite/cicd-cron/cicd-cron.rayci.yml

+      RAYCI_DISABLE_JAVA: "false"
+      RAYCI_WANDA_ALWAYS_REBUILD: "true"
+      JDK_SUFFIX: "-jdk"
+      HOSTTYPE: "{{matrix}}"


Bug: HOSTTYPE uses matrix placeholder without matrix defined

The manylinux-cibase-jdk-x86_64 step sets HOSTTYPE: "{{matrix}}" but this step has no matrix: field defined. The {{matrix}} placeholder won't be substituted and will remain as a literal string, causing the wanda build to use invalid values like quay.io/pypa/manylinux2014_{{matrix}} as the base image. This contrasts with the aarch64 variant on line 45 which correctly uses HOSTTYPE: "aarch64", and the x86_64 non-JDK variant on line 14 which uses HOSTTYPE: "x86_64".

ci/docker/manylinux-cibase.wanda.yaml

Signed-off-by: andrew <andrew@anyscale.com>

cursor · 2025-12-15T22:23:29Z

.buildkite/cicd-cron/cicd-cron.rayci.yml

+      - bazel run //ci/ray_ci/automation:copy_wanda_image --
+        --wanda-image-name manylinux-cibase-{{matrix}}
+        --destination-repository rayproject/manylinux2014
+        --tag-suffix -{{matrix}}


Bug: Push steps run in dry-run mode without uploading images

The bazel run //ci/ray_ci/automation:copy_wanda_image commands in both push steps don't include the --upload flag. Since copy_wanda_image.py defaults to dry-run mode when --upload is not provided (the upload parameter defaults to False), the pipeline will never actually copy images to Docker Hub despite the step labels saying "Push ... to Docker Hub". The script will just log "DRY RUN: Would copy..." and exit. This may be intentional for initial testing as noted in the PR description, but the nightly cron job won't upload any images if merged in this state.

Additional Locations (1)

.buildkite/cicd-cron/cicd-cron.rayci.yml#L71-L77

Signed-off-by: andrew <andrew@anyscale.com>

.buildkite/cicd-cron/cicd-cron.rayci.yml

Signed-off-by: andrew <andrew@anyscale.com>

CLAUDE.md

Signed-off-by: andrew <andrew@anyscale.com>

andrew-anyscale · 2025-12-16T02:58:24Z

Successfully built + uploaded as part of https://buildkite.com/ray-project/cicd-cron/builds/22/steps/canvas
See: https://hub.docker.com/r/rayproject/manylinux2014/tags

Now verifying separately in PR #59463

Adds new buildkite job for building + uploading a base `manylinux2014` image. Planning to build in flavors of with and without JDK included to help reduce image size when JDK is not needed. * Building is handled by a `wanda` builder step * Uploading to `rayproject/manylinux2014:{DATE}.{SHORT_COMMIT}-x86_64` and `rayproject/manylinux2014:{DATE}.{SHORT_COMMIT}-jdk-x86_64 is handled by a separate runner that calls the new `ci/ray_ci/automation/copy_wanda_image.py` * Runs by default without uploading for initial testing. Once passed with flag `--upload`, it will upload to the specified registry --------- Signed-off-by: andrew <andrew@anyscale.com> Co-authored-by: Lonnie Liu <95255098+aslonnie@users.noreply.github.com>

We now prebuild manylinux2014 with JDK as part of #59204 We can directly consume this, rather than rebuilding each time we need it --------- Signed-off-by: andrew <andrew@anyscale.com>

…59204) Adds new buildkite job for building + uploading a base `manylinux2014` image. Planning to build in flavors of with and without JDK included to help reduce image size when JDK is not needed. * Building is handled by a `wanda` builder step * Uploading to `rayproject/manylinux2014:{DATE}.{SHORT_COMMIT}-x86_64` and `rayproject/manylinux2014:{DATE}.{SHORT_COMMIT}-jdk-x86_64 is handled by a separate runner that calls the new `ci/ray_ci/automation/copy_wanda_image.py` * Runs by default without uploading for initial testing. Once passed with flag `--upload`, it will upload to the specified registry --------- Signed-off-by: andrew <andrew@anyscale.com> Co-authored-by: Lonnie Liu <95255098+aslonnie@users.noreply.github.com>

We now prebuild manylinux2014 with JDK as part of ray-project#59204 We can directly consume this, rather than rebuilding each time we need it --------- Signed-off-by: andrew <andrew@anyscale.com>

…59204) Adds new buildkite job for building + uploading a base `manylinux2014` image. Planning to build in flavors of with and without JDK included to help reduce image size when JDK is not needed. * Building is handled by a `wanda` builder step * Uploading to `rayproject/manylinux2014:{DATE}.{SHORT_COMMIT}-x86_64` and `rayproject/manylinux2014:{DATE}.{SHORT_COMMIT}-jdk-x86_64 is handled by a separate runner that calls the new `ci/ray_ci/automation/copy_wanda_image.py` * Runs by default without uploading for initial testing. Once passed with flag `--upload`, it will upload to the specified registry --------- Signed-off-by: andrew <andrew@anyscale.com> Co-authored-by: Lonnie Liu <95255098+aslonnie@users.noreply.github.com>

We now prebuild manylinux2014 with JDK as part of ray-project#59204 We can directly consume this, rather than rebuilding each time we need it --------- Signed-off-by: andrew <andrew@anyscale.com>

gemini-code-assist bot reviewed Dec 5, 2025

View reviewed changes

ci/base_images/utils/common.sh Outdated Show resolved Hide resolved

ci/base_images/utils/common.sh Outdated Show resolved Hide resolved

ci/base_images/utils/lib-docker.sh Outdated Show resolved Hide resolved

andrew-anyscale commented Dec 6, 2025

View reviewed changes

ci/base_images/nightly/build-manylinux.sh Outdated Show resolved Hide resolved

andrew-anyscale commented Dec 6, 2025

View reviewed changes

ci/base_images/nightly/build-manylinux.sh Outdated Show resolved Hide resolved

aslonnie self-requested a review December 8, 2025 22:32

andrew-anyscale commented Dec 8, 2025

View reviewed changes

.buildkite/cicd-cron/cicd-cron.rayci.yml Outdated Show resolved Hide resolved

andrew-anyscale commented Dec 9, 2025

View reviewed changes

ci/base_images/nightly/push-manylinux.sh Outdated Show resolved Hide resolved

andrew-anyscale commented Dec 9, 2025

View reviewed changes

andrew-anyscale force-pushed the andrewpollack/add-manylinux-nightly branch from acc0234 to d6e44c6 Compare December 9, 2025 18:06

andrew-anyscale marked this pull request as ready for review December 9, 2025 18:32

andrew-anyscale requested a review from a team as a code owner December 9, 2025 18:32

cursor bot reviewed Dec 9, 2025

View reviewed changes

ci/base_images/nightly/push-manylinux.sh Outdated Show resolved Hide resolved

ci/base_images/nightly/push-manylinux.sh Outdated Show resolved Hide resolved

ci/base_images/nightly/push-manylinux.sh Outdated Show resolved Hide resolved

andrew-anyscale changed the title ~~[draft][ci] Nightly for building+uploading manylinux baseimage~~ [ci] Nightly for building+uploading manylinux baseimage Dec 9, 2025

ray-gardener bot added the devprod label Dec 9, 2025

aslonnie reviewed Dec 10, 2025

View reviewed changes

andrew-anyscale added 14 commits December 11, 2025 01:40

[draft][ci] Nightly for building+uploading manylinux baseimage

22dcd86

Signed-off-by: andrew <andrew@anyscale.com>

cleanup: remove lib_docker

06f1a14

Signed-off-by: andrew <andrew@anyscale.com>

gemini suggestions

376597b

Signed-off-by: andrew <andrew@anyscale.com>

fix registry name

b4880bb

Signed-off-by: andrew <andrew@anyscale.com>

fixes

662dc62

Signed-off-by: andrew <andrew@anyscale.com>

feat(ci): add cicd-cron rayci definition

7292958

Scoped to just run build-many script. Signed-off-by: andrew <andrew@anyscale.com>

add doc about --upload

1c5e0dc

Signed-off-by: andrew <andrew@anyscale.com>

naming

5ab921c

Signed-off-by: andrew <andrew@anyscale.com>

move to separate build/push

86f59b0

Signed-off-by: andrew <andrew@anyscale.com>

cleanups

95a9cde

Signed-off-by: andrew <andrew@anyscale.com>

remove build-manylinux

2753490

We use wanda now Signed-off-by: andrew <andrew@anyscale.com>

add EOF

dd1d6c4

Signed-off-by: andrew <andrew@anyscale.com>

remove wanda install

e5ad8e9

Signed-off-by: andrew <andrew@anyscale.com>

andrew-anyscale added a commit to ray-project/rayci that referenced this pull request Dec 11, 2025

feat(wanda): add RAYCI_WANDA_ALWAYS_REBUILD

708510c

To be used in ray-project/ray#59204 Signed-off-by: andrew <andrew@anyscale.com>

add builders

a3a3def

Signed-off-by: andrew <andrew@anyscale.com>

andrew-anyscale force-pushed the andrewpollack/add-manylinux-nightly branch from b7aaa4a to a3a3def Compare December 15, 2025 21:56

cursor bot reviewed Dec 15, 2025

View reviewed changes

multi arch

7044502

Signed-off-by: andrew <andrew@anyscale.com>

cursor bot reviewed Dec 15, 2025

View reviewed changes

fixes

125c96f

Signed-off-by: andrew <andrew@anyscale.com>

cursor bot reviewed Dec 15, 2025

View reviewed changes

andrew-anyscale added 4 commits December 15, 2025 22:37

more updates yay

74a25a9

Signed-off-by: andrew <andrew@anyscale.com>

specify forge arch

e521cb2

Signed-off-by: andrew <andrew@anyscale.com>

temp test

0d97933

Signed-off-by: andrew <andrew@anyscale.com>

testing aarch

d8cde48

Signed-off-by: andrew <andrew@anyscale.com>

cursor bot reviewed Dec 16, 2025

View reviewed changes

.buildkite/cicd-cron/cicd-cron.rayci.yml Show resolved Hide resolved

send it

3835fc5

Signed-off-by: andrew <andrew@anyscale.com>

aslonnie self-requested a review December 16, 2025 02:39

aslonnie reviewed Dec 16, 2025

View reviewed changes

CLAUDE.md Outdated Show resolved Hide resolved

andrew-anyscale mentioned this pull request Dec 16, 2025

[ci]: use manylinux2014 nightly image #59463

Merged

remove accidental claude

d13afbd

Signed-off-by: andrew <andrew@anyscale.com>

aslonnie self-requested a review December 16, 2025 03:23

aslonnie added the go add ONLY when ready to merge, run all tests label Dec 16, 2025

Merge branch 'master' into andrewpollack/add-manylinux-nightly

e1ef003

aslonnie approved these changes Dec 16, 2025

View reviewed changes

aslonnie merged commit e211eed into master Dec 16, 2025
6 checks passed

aslonnie deleted the andrewpollack/add-manylinux-nightly branch December 16, 2025 06:59

Conversation

andrew-anyscale commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Dec 15, 2025

Choose a reason for hiding this comment

Bug: Missing matrix variable in JDK push step label

Uh oh!

Uh oh!

Uh oh!

cursor bot Dec 15, 2025

Choose a reason for hiding this comment

Bug: HOSTTYPE uses matrix placeholder without matrix defined

Uh oh!

Uh oh!

cursor bot Dec 15, 2025

Choose a reason for hiding this comment

Bug: Push steps run in dry-run mode without uploading images

Uh oh!

Uh oh!

Uh oh!

andrew-anyscale commented Dec 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

andrew-anyscale commented Dec 5, 2025 •

edited

Loading