Proposal: On-demand multi-machine Omega testing via GitHub PR labels

[cbegeman]

Problem

We currently have no way to run Omega ctests and the Polaris omega_pr suite across all supported machine/compiler combinations for a specific PR or branch on demand. The existing cron-scripts infrastructure gives us nightly testing, but the branch under test is fixed at install time — there is no mechanism for a developer to say "test this PR everywhere" without logging into each machine individually.

Proposed approach: PR label + API poll

A developer (or reviewer) applies a label such as test-on-hpc — or machine-specific variants like test-on-chrysalis, test-on-frontier — to an Omega PR.

On each supported machine, a cron job periodically polls the GitHub API for PRs carrying these labels. When it finds a newly labeled PR, it:

1. Records the PR's head.sha (so every machine tests the identical commit)
2. Checks out that commit and builds Omega for each supported compiler
3. Runs the Omega ctests and the Polaris omega_pr suite
4. Posts results back to the PR as a comment or commit status, including a link/path to logs
5. Swaps the label (e.g. test-on-hpc → hpc-tested) to mark completion

Per-machine state (last_tested_sha) keeps runs idempotent, so a re-push is required to retrigger.

Key property: the developer never signs into any machine — not to trigger the tests, and not to read the results.

Pros

• Zero per-run sign-in. Label from the browser; every machine picks it up and reports back.
• Trigger lives where review happens. No separate request repo or file to maintain — the PR is the request.
• Closes the loop. The same token that reads labels can post results, so failures surface in the PR thread rather than in a log file only the submitter can reach.
• Per-machine selectivity for free. Distinct labels target a subset of machines without new configuration.
• Unambiguous target. Polling returns head.sha, eliminating races with force-pushes.
• Built-in audit trail. Who requested testing, and when, is visible in the PR timeline.
• Reviewer visibility. Anyone looking at the PR can see that HPC testing was requested and what it found.

Cons

• Secrets on every machine. A GitHub token (fine-grained PAT or App installation token) must live in each machine's cron environment. Tokens expire, and renewal means signing back into each machine.
• Arbitrary code execution risk. Building a PR head runs unreviewed code under our HPC allocations. Needs a gate — honor the label only when applied by a maintainer, and be cautious with fork PRs.
• Polling is crude. Fixed cadence adds latency and requires DIY idempotency state, concurrency locks, and cleanup — effectively a hand-rolled version of webhook dispatch.
• Outbound network dependency. Requires reachable api.github.com from login nodes; proxy configuration is per-center toil.
• Cron fragility. Some centers disallow long-lived cron or require scrontab/systemd timers; jobs can die silently after password rotation. Needs a heartbeat to detect stoppage.
• Requires a PR. Pre-PR branch burn-in isn't supported.
• Two-repo coupling. Labels live on E3SM-Project/Omega; the runner lives in Polaris cron-scripts. Label naming and permissions must be coordinated.

Other approaches considered

Extend cron-scripts with a --branch argument. Minimal change, but cron only controls when a job runs, not what it tests — so the branch would have to be edited in each machine's crontab. Requires signing into every machine for every test run.

Git-based trigger (request file or dedicated request repo). Machines poll a small YAML request file and act on new entries. Fully token-free and expressive, but it has no return path — results are left on each machine's filesystem, so the developer must sign in to read them. Adding a post-back mechanism reintroduces the credential distribution it was designed to avoid. Committing requests to Polaris main also adds PR/merge friction and pollutes history.

Ref-based trigger. Push a ref like test-request/chrysalis/my-feature; machines poll with git ls-remote. Token-free, needs no new repo, and works before a PR exists — but shares the git-based trigger's missing return path and gives reviewers no visibility.

Self-hosted GitHub Actions runners. Registered on each machine, driven by webhooks with workflow_dispatch branch inputs. Technically the strongest option: event-driven rather than polled, with log streaming and PR checks integration out of the box. Blocked or discouraged by policy at some centers, and carries a heavier one-time setup burden. As far as I can tell, this is what EAMxx uses https://github.com/E3SM-Project/E3SM/blob/master/.github/workflows/eamxx-sa-testing.yml?utm_source=chatgpt.com

SSH fan-out from a single host. A driver script loops over machines over SSH. Blocked in practice by MFA at most DOE facilities.

Manual documented recipe. A helper script run interactively. Requires signing into every machine, every time.

Cons mitigated by the label/API-poll approach

Relative to the alternatives, this approach specifically resolves:

• The missing return path (git- and ref-based triggers) — results post directly to the PR.
• Per-run sign-in on every machine (--branch in crontab, manual recipe) — triggering is a browser click.
• MFA blocking remote orchestration (SSH fan-out) — polling is outbound-only, so no interactive auth is needed.
• Push/merge friction and main history pollution (request file in main) — no commit is required to request a run.
• Zero reviewer visibility (all git-based variants) — the request and its outcome sit in the PR timeline.
• Site policy restrictions on self-hosted runners — this needs only cron and outbound HTTPS, both already available where cron-scripts runs today.

It does not mitigate cron fragility or polling latency, both of which only self-hosted runners fully address.

Discussion questions

1. Is distributing a GitHub token to each machine acceptable, or is the token-free ref-based trigger preferable despite losing the return path?
2. Are self-hosted runners viable at any of our target centers? If so, should we treat label polling as a fallback only for machines where they're disallowed?

[cbegeman]

GitHub token requirements for the label/API-poll approach

What the token must be able to do

The cron job on each machine performs three distinct GitHub operations, and each drives a scope requirement:

Operation	API endpoint	Permission needed
Find labeled PRs and their head.sha	GET /repos/E3SM-Project/Omega/pulls (or /issues with labels= filter)	Pull requests: read
Determine who applied the label (maintainer gate)	GET /repos/.../issues/{n}/timeline	Issues: read
Post results back as a PR comment	POST /repos/.../issues/{n}/comments	Issues: write
(alternative) Post results as a commit status/check	POST /repos/.../statuses/{sha}	Commit statuses: write
Swap test-on-hpc → hpc-tested to mark completion	DELETE/POST on /issues/{n}/labels	Issues: write

If E3SM-Project/Omega is public, the read half needs no token at all — but unauthenticated requests are rate-limited to 60/hr per IP, which is shared across everyone on a login node. Authentication raises this to 5,000/hr, so a token is effectively mandatory even for read-only polling.

Token type: three options

1. Fine-grained PAT (simplest)

• Owner: a maintainer's account, or better, a dedicated bot/service account.
• Repository access: E3SM-Project/Omega only.
• Permissions: Pull requests: Read, Issues: Read and write.
• Expiry: maximum 1 year; many org policies force 90 days or less.
• Downside: renewal requires signing into every machine to replace the secret — the exact cost this design was meant to avoid, just amortized.

2. GitHub App installation token (recommended for a shared setup)

• Register a "Polaris HPC Test Runner" App installed on E3SM-Project/Omega.
• Each machine stores the App's private key (a PEM file), not a token.
• The cron script mints a short-lived JWT, exchanges it for an installation token valid 1 hour, and discards it.
• Advantages: the long-lived secret is a private key you control and can revoke centrally; tokens on disk are ephemeral; actions appear as the App, not a person; permissions are declared once at the App level.
• Cost: ~60 extra lines of token-minting logic and a JWT-signing dependency (PyJWT + cryptography, both available in the Polaris environment).

3. Classic PAT — not recommended

repo scope grants write access to every repository the owner can reach. Far too broad for a credential sitting on multiple shared HPC filesystems.

Storage and handling on each machine

• Path convention: $HOME/.config/polaris/github_token (or github_app.pem), mode 0600, owned by the cron user.
• Never place the secret in /lcrc/group/... or any group-readable project space — those are readable by the whole allocation.
• Do not put the token in the crontab line or in any script under cron-scripts; read it from the file at runtime.
• Export it only into the environment of the specific curl/Python call that needs it, not the whole build/test job — otherwise it leaks into the environment of every compile and run step, and into any env-dumping log.
• Scrub it from logs. Anything using set -x will echo it; guard those sections with set +x.
• Consider a per-machine token (or per-machine App installation, if feasible) so a compromise on one system can be revoked without disrupting the others.

Operational concerns

• Expiry is the main failure mode. A silently expired token turns into "the machine stopped testing and nobody noticed." Add a heartbeat: on each poll, record a timestamp and the HTTP status to a status file, and treat a 401 as a loud, distinguishable failure rather than "no PRs found."
• Pre-expiry warning. GET /rate_limit or the PAT metadata endpoint can be checked periodically; alert (email from the cron job) some weeks before the token lapses.
• Read/write split. You can reduce blast radius by using a token-free anonymous read for polling and reserving an authenticated token solely for posting results — but the 60/hr rate limit makes this impractical at any useful cadence.
• Fork PRs. The token's permissions apply to the base repository, so posting comments on fork-originated PRs works fine. The risk there is not the token but building unreviewed code — handle that with the maintainer-label gate, not with token scoping.
• Audit. Fine-grained PAT and App activity is visible in the org audit log, which helps if you need to trace an unexpected label change or comment.

Recommendation

Use a GitHub App with Pull requests: Read and Issues: Read and write, scoped to E3SM-Project/Omega, with the private key stored at 0600 in each cron user's home directory. It costs a bit more up front than a PAT, but it avoids the recurring "sign into every machine to rotate the token" chore that would otherwise undermine the central premise of this approach — and it makes revocation a single action rather than N.

If you want a faster proof of concept, start with a fine-grained PAT on a bot account, accept the rotation burden, and migrate to an App once the polling logic in launch_all.sh is settled.

[cbegeman]

Reuse of existing cron-scripts machinery

Quite a lot — the label/API-poll approach is best understood as a new trigger front-end bolted onto the existing dispatch back-end, not a parallel system.

Reused essentially unchanged

• machines/config_*.sh — the per-machine definitions (account, partition, compilers, module setup, scratch paths) for chrysalis, frontier, pm-cpu, pm-gpu, aurora are exactly what a PR-triggered run needs. config_machine.sh already provides the machine-detection/dispatch indirection.
• tasks/omega_cdash/job_<machine>_omega_*.{sbatch,pbs} — the batch submission wrappers are agnostic to why they were launched. They need to accept a commit SHA rather than assuming a default branch, but the resource requests, module loads, and job plumbing carry over verbatim.
• tasks/omega_cdash/run_omega_cdash.sh — the actual build + ctest + omega_pr execution logic is the same work regardless of trigger.
• CDash submission (CTestScript.txt) — results can still be posted to CDash; the PR comment becomes an additional reporting channel with a link into the CDash build, which is a nice division of labor (summary in the PR, detail in CDash).
• launch_all.sh's fan-out structure — the loop over machines/tasks and the log-directory conventions stay.

What has to be added

• A polling/trigger layer in front of launch_all.sh — GitHub API query, maintainer-label gate, head.sha extraction, per-machine last_tested_sha state, and a concurrency lock. This is genuinely new and is the bulk of the work.
• Branch/SHA parameterization threaded from the trigger through launch_all.sh → launch_omega_cdash.sh → the job scripts. Currently the Omega checkout is implicit; it becomes an explicit input. This is the same change option (1) would have required, so it's not wasted effort under any design.
• A results post-back step — new, and the piece with no analogue in the current scripts.
• Token handling and heartbeat/status reporting — new.

Rough proportion

Perhaps 70–80% of the per-machine execution path is reused as-is, with the new code concentrated at the two ends: trigger resolution before dispatch, and result reporting after. The machine configs — which are the genuinely expensive, site-specific, hard-to-get-right part — require no rework beyond accepting a SHA.

A useful consequence

Because the trigger layer is separable, the same back-end can serve nightly cron and on-demand PR testing, with the trigger step simply resolving to "the default branch" in the nightly case. That also means a ref-based or file-based trigger could be added later as an alternative front-end without touching anything under tasks/ or machines/ — so the choice of trigger channel is not a one-way door.

[rljacob]

CC @bartgol @mahf708 . We would want this for E3SM in general?

[rljacob]

Also cc @jgfouca @ndkeen @amametjanov

[cbegeman]

Thanks for socializing @rljacob! I saw the github workflow for EAMxx from @jgfouca and it could be that some variant of that would also work for Omega ctesting. I started going down this route partly because it builds on @grnydawn's work on https://github.com/E3SM-Project/polaris/tree/main/cron-scripts, #530 (comment), which covers both ctests and our standalone whole-component tests with Polaris

[grnydawn]

This would be really nice to have. A few concerns:

• Cron fragility. Crontabs don't reliably survive reboots.
• Job submission limits. Some centers cap concurrent jobs.

These may or may not be issues depending on the amount of workload.

[mahf708]

My 2 exact cents:

1. I think per-hpc testing is overrated and I don't know if it is worth running it for every PR, I think running it nightly suffices. I would even argue for running it like once a week.
2. I would focus on running the tests more interactively on cpu (preferrably on free github-hosted action runners) and gpu (either paid github-hosted or some self-hosted action runners --- eamxx does the latter)

[bartgol]

I agree with Naser that extensive machine-specific testing on each PR may be too much. A containerized version of a handful of versions of the code may suffice. Maybe a gnu and intel builds (both CPU), and a CUDA build.

Btw, the eamxx CI runs on gh runners hosted inside SNL machines, and is available for ALL e3sm-project projects. The only catch is that there are restrictions on what job can run: if there is a commit by someone NOT in the snl-testing gh team, there MUST be an approval by a member of the snl-testing gh team. Only folks that have access to the SNL machines hosting the runners can be in that team.

If this is an option that works for you, you could use those runners. Maciej is already a SNL person, and if some other Omega folks got a SNL collaborator account, you could be good to go.

Of course, if you don't care about GPU testing for PRs, and your tests don't require baselines, you couldshould also consider running on free gh runners.

[xylar]

Thanks @mahf708 and @bartgol — I appreciate your thoughts but I want to explain why we've landed somewhere different for Omega specifically.

Our recent experience has been that not testing each PR everywhere is the expensive option. When several PRs land between full test runs and an unexpected non-BFB change shows up, untangling which PR caused it (and on which machine/compiler) has repeatedly cost us far more time than the testing would have. We've also had nightly runs surface non-BFB differences or outright build failures on compilers that the PR-time subset didn't cover — again, after several PRs were already in. At Omega's current stage, every corner we've tried to cut has slowed us down rather than sped us up. That may well change as the code matures, and I'd expect us to relax the cadence then.

Worth noting the proposal is on-demand rather than automatic: the label means we can scope which machines a given PR goes to, so the cost isn't a fixed tax on every PR.

Luca, thanks for the offer of the SNL-hosted runners — I don't think it works for us in practice, since not enough of our team are at Sandia and every PR from the rest of us would need his approval to run.

We do need GPU testing and coverage of all supported compilers, and I feel we need it on the real machines at realistic problem sizes. CI-sized cases in containers won't surface the class of problem that's actually been biting us. I can see the value in investigating containerized gnu/intel/CUDA builds on free runners as a fast first tier — quick build and unit-test feedback before the heavier runs — just not as a replacement for them.

[mahf708]

Hmmmm, this is surprising, but maybe not...

Of course, you know your codebase better and so we only offer outside perspective... but it is a troubling sign that you're facing these machine-specific issues, hinting at potentially fundamental code/logic/algorithmic issues. I vaguely remember having to deal with some MPAS issues across machines when I added the analysis member codebase (in my massive fme aigo branch), so perhaps you're inherting similar issues there? 🤔

Anyway, I think you should also consider the drudgerous churn needed to maintain cross-machine tests versus having representative unit-like/property-esque tests that will capture a lot of issues via small cheap tests instead of several machine-specific launches. So investing in getting to a stage where you have representative tests may well be the solution, but you need to start somewhere first and fast to stop the time bleeding...

[xylar]

I don't see any evidence that differences in non-BFB behavior that we're seeing across E3SM supported machines is in any way inherited from MPAS. Other developers can likely speak to the details better but it seems instead to relate to fiddly details with Kokkos.

[bartgol]

I understand your concerns, and as Naser said, you guys own the code, and decide how to handle testing.

That said, as a personal opinion, I think that master is not necessarily the production branch. Each science campaign branches off master, using a specific tag/branch, testing it, ensuring no issue show up at the res needed for the science campaign, and only THEN run. If there is an issue that shows up on a certain machine with a certain config, they are free to revert the PR that introduced it, if waiting for the fix is not doable due to time constraints. So if an issue only shows up at scale and/or on a specific machine, I find it acceptable to have the issue in master, while we work on a fix. E3SM does not have a "develop" branch, but master is just that, for all practical purposes. The official branches are maint-X.Y.Z, and those do have to be fully bug free.

I personally find PR testing important for structural issues, and nightly testing important for targeted issues. I am fine with EAMxx having an issue in master for a specific machine/config. But maybe that's because I don't usually have to manage long/complex science campaigns. :)

tl;dr Your approach is completely understandable. EAMxx may have a different view (or maybe different needs), but that's ok.