feat(documentation): optional GPU upgrade pass, publish-then-upgrade - #71
Merged
Conversation
For control-toolbox/OptimalControl.jl#885 part 2. `docs/src/solve/gpu.md` is now fully executable there (#941), and a feasibility probe on `occidata` (2026-09-02) confirmed all four preconditions: the npm registry is reachable, `CUDA.functional() = true` with a real device (GeForce GTX 1080 Ti), a real `:gpu` solve completes (205s), and the node can push to the repo via the same mechanism live deployment already uses (`GITHUB_TOKEN` over HTTPS -- Documenter's own `deployconfig.jl` never picks SSH here, since `DOCUMENTER_KEY` isn't passed by that caller). A full docs build there took 2265s. Design: *publish then upgrade*, not probe-then-fallback. The existing `build` job is untouched and always runs first -- so docs publishing never depends on a self-hosted box being online, reachable, or fast. A new `build-gpu` job runs only once `build` has already deployed (`needs: build`) and, with `continue-on-error: true`, can never fail the workflow: on success it redeploys the same site with real GPU numbers; on failure, or if the runner is queued behind other work and the `gpu_timeout_minutes` budget runs out, the site `build` already published is simply left as is. New inputs, both optional and both default to reproducing today's behaviour exactly for every existing caller: - `gpu_runner` (default `''`): JSON array of self-hosted runner labels, e.g. `'["occidata"]'`. Empty disables `build-gpu` entirely -- it shows as `skipped` in the run graph, at no cost to callers that never set it. - `gpu_timeout_minutes` (default `120`): budget for a *cold* self-hosted build. Self-hosted depots persist across runs and get purged by maintenance jobs (occidata's Monday 02:30 UTC cache purge), so the first build after one is the number to budget for, not a warm rebuild. `build-gpu` also carries the self-hosted adaptations `ci.yml` already established and this file's own `build` job (GitHub-hosted) has no need of: - resolve the real Julia depot path instead of assuming `~/.julia` (`setup-julia` does not set `JULIA_DEPOT_PATH`, and self-hosted depots vary by machine); - redirect `TMPDIR` off the node's periodically-wiped `/tmp` before any Julia step runs -- a docs build is longer than a test run, so it will almost certainly cross a wipe boundary; - purge-and-retry the compiled cache once on failure, since a persistent depot can drift out of sync with a different branch's Manifest. `build-gpu` only runs for `push`/`tag` triggers, never for `pull_request` -- PR-preview docs builds stay GitHub-hosted-only and fast, matching what #885 itself proposed. No change to the `build` job. Every existing caller (all packages using this reusable workflow) is unaffected: `gpu_runner` defaults to `''`, so `build-gpu`'s `if` is false and it shows `skipped`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of control-toolbox/OptimalControl.jl#885. Part 1 (making
docs/src/solve/gpu.mdfully executable) shipped as OptimalControl.jl#941. This PR adds the reusable-workflow plumbing for a GPU-backed docs build; the caller-side PR on OptimalControl.jl follows this one.Feasibility, measured 2026-09-02 on
occidataBefore touching this file, a throwaway probe workflow answered all four open questions:
npm install)HTTP 200CUDA.functional() = true, 1× GeForce GTX 1080 Ti (10.9 GiB), CUDA 12.9 toolchain; a realsolve(ocp, :gpu; grid_size=1000)completed in 205sdocs/make.jlbuild take theregpu_timeout_minutesdefaultOn the last point:
Documenter's owndeployconfig.jlcomputesauthentication_method(::GitHubActions) = env_nonempty("DOCUMENTER_KEY") ? SSH : HTTPS, andDOCUMENTER_KEYis not passed by OptimalControl.jl'sDocumentation.ymlcaller — so live deployment is HTTPS via the ephemeralGITHUB_TOKEN, never SSH. A first probe attempt pushed via an SSH deploy key instead (scoped toct-registry, unrelated) and was correctly denied; a second, corrected probe pushed viaGITHUB_TOKEN/HTTPS and succeeded — verified independently on GitHub. The job's ownSet up joblog also showedContents: writein the ambientGITHUB_TOKENpermissions before any step ran.Design — publish then upgrade, not probe-then-fallback
Two structural facts rule out the naive
runs-on: [occidata]swap:timeout-minutesdoes not bound a queue wait. It starts once a runner picks the job up; a job with no runner available just sits queued (GitHub's self-hosted cap is ~24h), so a timeout alone gives no fallback.docs/make.jlends indeploydocs(...), so the job that builds is the job that deploys — they cannot be split across runners without surgery Documenter doesn't support.So instead of attempting GPU and falling back to CPU, this PR always runs the existing
buildjob first, unchanged, and adds a second job,build-gpu, that runs only oncebuildhas already deployed (needs: build) and, withcontinue-on-error: true, can never fail the workflow. On success it redeploys the same site with real GPU output; on failure, or if the self-hosted runner is queued past its budget, the sitebuildalready published is simply left as is. Docs publishing is never on the critical path of a self-hosted box being online, reachable, or fast — the worst case when the GPU box is down (occidata is a SLURM job, watched by only a daily cron) is that dev docs keep showing the CPU story a while longer, exactly today's situation.What's in this PR
Two new
workflow_callinputs, both optional, both defaulting to reproducing today's behaviour exactly for every existing caller:gpu_runner(default''): JSON array of self-hosted runner labels, e.g.'["occidata"]'. Empty ⇒build-gpu'sifis false, it showsskippedin the run graph — zero cost to callers that never set it (CTBase, CTModels.jl, CTDirect.jl, CTFlows.jl, CTSolvers, CTParser.jl, CTLie all keep exactly today's single-job workflow).gpu_timeout_minutes(default120): budget for a cold self-hosted build — self-hosted depots persist across runs and get purged by maintenance jobs (occidata's own Monday 02:30 UTC cache purge), so the number to budget for is a cold compile, not the 2265s warm run measured above.build-gpuonly runs forpush/tagtriggers (github.event_name != 'pull_request') — PR-preview docs builds stay GitHub-hosted-only and fast, matching what OptimalControl.jl#885 itself proposed.build-gpucarries three self-hosted adaptations already established inci.yml, which the GitHub-hostedbuildjob has no need of:~/.julia(setup-juliadoesn't setJULIA_DEPOT_PATH, and it varies by self-hosted machine);TMPDIRoff the node's periodically-wiped/tmpbefore any Julia step — occidata wipes it hourly atHH:01, and a docs build is long enough to almost certainly cross that boundary (ci.yml, commit 2f00b67);No change to the
buildjob — verified by diff.Not in this PR
Documentation.ymlpassinggpu_runner: '["occidata"]'— that's the caller-side PR, opened once this merges.Handbook/WORKFLOWS.mdupdates (thedocumentation.ymlrow's inputs table, and the runner-fleet corrections the probe also surfaced:kktis no longer a registered org runner at all, and there's an undocumented third self-hosted box,pinwheel, which is AMD — not a candidate for this or any:gpuCI, since the whole stack fromExaModelsdown is CUDA-only).🤖 Generated with Claude Code