CI: maintainer-triggered GPU test suites via PR comment - #115
Merged
Conversation
…d on comments on prs made by either admin or maintainer. posts the results as sticky comments in the pr Signed-off-by: aviv ron <rona@il.ibm.com>
aviv1ron1
marked this pull request as ready for review
August 5, 2026 17:03
aviv1ron1
requested review from
antonpibm,
freunda and
yairallouche
as code owners
August 5, 2026 17:03
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
yairallouche
approved these changes
Aug 6, 2026
aviv1ron1
added a commit
that referenced
this pull request
Aug 6, 2026
Brings in the self-hosted runner GPU test workflow (#115): comment-driven test runs gated on admin/maintainer role, with results posted as sticky PR comments. Conflict in .github/workflows/gpu-tests.yaml resolved in favor of main. The asr side's setup-uv + 'uv sync --extra audio' step is obsolete under main's design, where the helper scripts and environment are baked into the runner image at /opt/gsw rather than installed per run. Audio coverage is preserved as a first-class suite scope selected by marker. Signed-off-by: aviv ron <rona@il.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a maintainer-triggered path to run the GPU-dependent test suites on
self-hosted runner, launched from a PR comment and
reported back onto the PR.
Replaces the earlier placeholder
gpu-tests.yaml, which assumed aGitHub-hosted runner.
Usage
A maintainer comments one of these on a PR:
/gpu-test/gpu-test-shorttests/vllm/+tests/integration//gpu-test-audio-m "audio and not deep", across all suites/gpu-test-multi/gpu-test-devtests/vllm/test_single_switch.py- minimal to test the automation itselfBoth supported vLLM lines run as separate matrix legs (
vllm19,vllm20),since their dependency groups conflict and can't share an environment.
An unrecognised
/gpu-test-*is declined with a reply listing the valid ones.Who can trigger it
Two independent gates:
pull_requesttrigger.workflow_dispatchonly, so a pull request —including from a fork — can never start a run by itself.
check_role.shrejects any actor without the Maintain orAdmin role. It runs inside
gpu-tests.yamlitself, so it also covers adirect dispatch from the Actions tab, which needs only write access.
The comment handler is convenience only;
gpu-tests.yamlre-checks the role.What appears on the PR
Per matrix leg, per scope: a commit status, a sticky comment carrying
pytest's own counts (
2 failed, 1338 passed, …) and a log excerpt, and a logartifact. Each scope reports under its own status context
(
gpu-tests (vllm19),gpu-tests (vllm19-dev), …), so a passing quick checkcan't overwrite — and hide — a failing full run.
Files
.github/workflows/gpu-tests.yaml.github/workflows/gpu-test-command.yaml.github/scripts/gpu_test_command.sh.github/scripts/check_role.shThe two scripts are checked in rather than supplied by the runner because the
comment listener runs on a GitHub-hosted runner. Both contain only GitHub API
calls — no credentials and no infrastructure configuration. This workflow holds
no cluster access of any kind.
Notes for reviewers
gpu-test-command.yamlonly fires from the default branch. The commandsstay inert until this merges to
main; they can't be exercised on the PRbranch.
permissions:is scoped tostatuses: write,pull-requests: write,contents: read.