WIP: stabilize nightly builds + tests #359
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jameslamb
added
non-breaking
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
8 tasks
Collaborator
|
@jameslamb Please let me know if it is in a good state to be moved to review |
Member
Author
Not yet, will |
8 tasks
Member
Author
|
/ok to test |
rapids-bot bot
pushed a commit
that referenced
this pull request
Sep 22, 2025
I'd missed `cuopt` in rapidsai/shared-workflows#376, and also want to pull these changes off of #359 to shrink the diff there. ## Issue Helps with #122 (via making #359 a bit easier to review) Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Ramakrishnap (https://github.com/rgsl888prabhu) URL: #407
Member
Author
rapids-bot bot
pushed a commit
that referenced
this pull request
Sep 24, 2025
Replaces #359 (my more-complicated earlier attempt at this) This project runs nightly builds and tests on a cron schedule: https://github.com/NVIDIA/cuopt/blob/36a6a1c0edf42cec2cf07c6be3f16531f33515de/.github/workflows/nightly.yaml#L1-L6 Tests need to wait for builds to finish, and that's currently done with some shell scripts that hit the GitHub API, using a mix of `sleep` and polling. This has sometimes resulted in nightly failures (network errors, timeouts, etc.). This PR proposes reducing the risk of such failures by moving that logic into GitHub Actions configuration directly, specifically: * making `build.yaml` trigger `test.yaml` with the GitHub CLI **only after all package builds and publishing have finished** ## Issue Contributes to #122 ## Notes for Reviewers ### How I tested this I manually triggered this run of the "Trigger Nightly cuOpt Pipeline": https://github.com/NVIDIA/cuopt/actions/runs/17935159871 Which triggered this `build` run: https://github.com/NVIDIA/cuopt/actions/runs/17935161536 Which triggered this `test` run: https://github.com/NVIDIA/cuopt/actions/runs/17936474025 Things look ok to me! The `test` run was triggered until after all the relevant package builds and uploads were done, and BEFORE the docker image builds were done (as intended, to not be delayed waiting on them). There are some test failures from artifact-downloading, like this: ```text [rapids-github-run-id] Querying the GitHub API to determine relevant run of 'build.yaml'. Downloading and decompressing cuopt_wheel_python_cuopt_server_cu12_py312_x86_64 from Run ID 17936253863 into /tmp/tmp.pqrBXIhMlP ``` But I think they'll be fixed by merging #409 And the naming changes for the image builds look good 😁 <img width="317" height="203" alt="image" src="https://github.com/user-attachments/assets/31bac7bd-1c4d-4c31-9ce9-9863778c2e89" /> Authors: - James Lamb (https://github.com/jameslamb) Approvers: - Ramakrishnap (https://github.com/rgsl888prabhu) URL: #408
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This project runs nightly builds and tests on a cron schedule:
cuopt/.github/workflows/nightly.yaml
Lines 1 to 6 in 36a6a1c
Tests need to wait for builds to finish, and that's currently done with some shell scripts that hit the GitHub API, using a mix of
sleepand polling.This has sometimes resulted in nightly failures (network errors, timeouts, etc.). This PR proposes reducing the risk of such failures by moving that logic into GitHub Actions configuration directly, specifically:
build.yamlandtest.yamlre-usable workflows and havingnightly.yamlcall them withuses:test.yamluntilbuild.yamlpasses usingneeds:Issue
Closes #122
Checklist