Skip to content

CI: maintainer-triggered GPU test suites via PR comment - #115

Merged
yairallouche merged 1 commit into
mainfrom
test_automation
Aug 6, 2026
Merged

CI: maintainer-triggered GPU test suites via PR comment#115
yairallouche merged 1 commit into
mainfrom
test_automation

Conversation

@aviv1ron1

Copy link
Copy Markdown
Collaborator

What

Adds a maintainer-triggered path to run the GPU-dependent test suites on
self-hosted runner, launched from a PR comment and
reported back onto the PR.

Replaces the earlier placeholder gpu-tests.yaml, which assumed a
GitHub-hosted runner.

Usage

A maintainer comments one of these on a PR:

command scope rough cost
/gpu-test all five suites hours
/gpu-test-short tests/vllm/ + tests/integration/ tens of minutes
/gpu-test-audio -m "audio and not deep", across all suites minutes
/gpu-test-multi all five suites on 2 GPUs — runs the TP/PP tests that are otherwise skipped hours
/gpu-test-dev tests/vllm/test_single_switch.py - minimal to test the automation itself a couple of minutes

Both supported vLLM lines run as separate matrix legs (vllm19, vllm20),
since their dependency groups conflict and can't share an environment.

An unrecognised /gpu-test-* is declined with a reply listing the valid ones.

Who can trigger it

Two independent gates:

  • No pull_request trigger. workflow_dispatch only, so a pull request —
    including from a fork — can never start a run by itself.
  • Role check. check_role.sh rejects any actor without the Maintain or
    Admin role. It runs inside gpu-tests.yaml itself, so it also covers a
    direct dispatch from the Actions tab, which needs only write access.

The comment handler is convenience only; gpu-tests.yaml re-checks the role.

What appears on the PR

Per matrix leg, per scope: a commit status, a sticky comment carrying
pytest's own counts (2 failed, 1338 passed, …) and a log excerpt, and a log
artifact
. Each scope reports under its own status context
(gpu-tests (vllm19), gpu-tests (vllm19-dev), …), so a passing quick check
can't overwrite — and hide — a failing full run.

Files

file purpose
.github/workflows/gpu-tests.yaml the run itself: role gate, matrix, reporting
.github/workflows/gpu-test-command.yaml comment listener; dispatches the above
.github/scripts/gpu_test_command.sh parses the command, checks the role, dispatches
.github/scripts/check_role.sh the Maintain/Admin gate

The two scripts are checked in rather than supplied by the runner because the
comment listener runs on a GitHub-hosted runner. Both contain only GitHub API
calls — no credentials and no infrastructure configuration. This workflow holds
no cluster access of any kind.

Notes for reviewers

  • gpu-test-command.yaml only fires from the default branch. The commands
    stay inert until this merges to main; they can't be exercised on the PR
    branch.
  • permissions: is scoped to statuses: write, pull-requests: write,
    contents: read.

…d on comments on prs made by either admin or maintainer. posts the results as sticky comments in the pr

Signed-off-by: aviv ron <rona@il.ibm.com>
@aviv1ron1
aviv1ron1 marked this pull request as ready for review August 5, 2026 17:03
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@yairallouche
yairallouche merged commit 91bf799 into main Aug 6, 2026
4 checks passed
aviv1ron1 added a commit that referenced this pull request Aug 6, 2026
Brings in the self-hosted runner GPU test workflow (#115): comment-driven
test runs gated on admin/maintainer role, with results posted as sticky PR
comments.

Conflict in .github/workflows/gpu-tests.yaml resolved in favor of main.
The asr side's setup-uv + 'uv sync --extra audio' step is obsolete under
main's design, where the helper scripts and environment are baked into the
runner image at /opt/gsw rather than installed per run. Audio coverage is
preserved as a first-class suite scope selected by marker.

Signed-off-by: aviv ron <rona@il.ibm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants