Skip to content

feat(documentation): optional GPU upgrade pass, publish-then-upgrade - #71

Merged
ocots merged 1 commit into
mainfrom
feat/gpu-docs-build
Sep 2, 2026
Merged

feat(documentation): optional GPU upgrade pass, publish-then-upgrade#71
ocots merged 1 commit into
mainfrom
feat/gpu-docs-build

Conversation

@ocots

@ocots ocots commented Sep 2, 2026

Copy link
Copy Markdown
Member

Part 2 of control-toolbox/OptimalControl.jl#885. Part 1 (making docs/src/solve/gpu.md fully executable) shipped as OptimalControl.jl#941. This PR adds the reusable-workflow plumbing for a GPU-backed docs build; the caller-side PR on OptimalControl.jl follows this one.

Feasibility, measured 2026-09-02 on occidata

Before touching this file, a throwaway probe workflow answered all four open questions:

check result
npm registry reachable (DocumenterVitepress bundles its own node but still runs npm install) HTTP 200
does CUDA 6.2 + CUDSS 0.7 — the only combination that resolves in the docs environment — work on the real driver yes: CUDA.functional() = true, 1× GeForce GTX 1080 Ti (10.9 GiB), CUDA 12.9 toolchain; a real solve(ocp, :gpu; grid_size=1000) completed in 205s
how long does a full docs/make.jl build take there 2265s (~38 min) — sets this PR's gpu_timeout_minutes default
can the node push via the mechanism deployment actually uses yes — confirmed the hard way (see below)

On the last point: Documenter's own deployconfig.jl computes authentication_method(::GitHubActions) = env_nonempty("DOCUMENTER_KEY") ? SSH : HTTPS, and DOCUMENTER_KEY is not passed by OptimalControl.jl's Documentation.yml caller — so live deployment is HTTPS via the ephemeral GITHUB_TOKEN, never SSH. A first probe attempt pushed via an SSH deploy key instead (scoped to ct-registry, unrelated) and was correctly denied; a second, corrected probe pushed via GITHUB_TOKEN/HTTPS and succeeded — verified independently on GitHub. The job's own Set up job log also showed Contents: write in the ambient GITHUB_TOKEN permissions before any step ran.

Design — publish then upgrade, not probe-then-fallback

Two structural facts rule out the naive runs-on: [occidata] swap:

  1. timeout-minutes does not bound a queue wait. It starts once a runner picks the job up; a job with no runner available just sits queued (GitHub's self-hosted cap is ~24h), so a timeout alone gives no fallback.
  2. Deploy is inside the build. docs/make.jl ends in deploydocs(...), so the job that builds is the job that deploys — they cannot be split across runners without surgery Documenter doesn't support.

So instead of attempting GPU and falling back to CPU, this PR always runs the existing build job first, unchanged, and adds a second job, build-gpu, that runs only once build has already deployed (needs: build) and, with continue-on-error: true, can never fail the workflow. On success it redeploys the same site with real GPU output; on failure, or if the self-hosted runner is queued past its budget, the site build already published is simply left as is. Docs publishing is never on the critical path of a self-hosted box being online, reachable, or fast — the worst case when the GPU box is down (occidata is a SLURM job, watched by only a daily cron) is that dev docs keep showing the CPU story a while longer, exactly today's situation.

What's in this PR

Two new workflow_call inputs, both optional, both defaulting to reproducing today's behaviour exactly for every existing caller:

  • gpu_runner (default ''): JSON array of self-hosted runner labels, e.g. '["occidata"]'. Empty ⇒ build-gpu's if is false, it shows skipped in the run graph — zero cost to callers that never set it (CTBase, CTModels.jl, CTDirect.jl, CTFlows.jl, CTSolvers, CTParser.jl, CTLie all keep exactly today's single-job workflow).
  • gpu_timeout_minutes (default 120): budget for a cold self-hosted build — self-hosted depots persist across runs and get purged by maintenance jobs (occidata's own Monday 02:30 UTC cache purge), so the number to budget for is a cold compile, not the 2265s warm run measured above.

build-gpu only runs for push/tag triggers (github.event_name != 'pull_request') — PR-preview docs builds stay GitHub-hosted-only and fast, matching what OptimalControl.jl#885 itself proposed.

build-gpu carries three self-hosted adaptations already established in ci.yml, which the GitHub-hosted build job has no need of:

  • resolve the real Julia depot path instead of assuming ~/.julia (setup-julia doesn't set JULIA_DEPOT_PATH, and it varies by self-hosted machine);
  • redirect TMPDIR off the node's periodically-wiped /tmp before any Julia step — occidata wipes it hourly at HH:01, and a docs build is long enough to almost certainly cross that boundary (ci.yml, commit 2f00b67);
  • purge-and-retry the compiled cache once on failure, since a persistent depot can drift out of sync with a different branch's Manifest.

No change to the build job — verified by diff.

Not in this PR

  • OptimalControl.jl's Documentation.yml passing gpu_runner: '["occidata"]' — that's the caller-side PR, opened once this merges.
  • Handbook/WORKFLOWS.md updates (the documentation.yml row's inputs table, and the runner-fleet corrections the probe also surfaced: kkt is no longer a registered org runner at all, and there's an undocumented third self-hosted box, pinwheel, which is AMD — not a candidate for this or any :gpu CI, since the whole stack from ExaModels down is CUDA-only).

🤖 Generated with Claude Code

For control-toolbox/OptimalControl.jl#885 part 2. `docs/src/solve/gpu.md`
is now fully executable there (#941), and a feasibility probe on
`occidata` (2026-09-02) confirmed all four preconditions: the npm
registry is reachable, `CUDA.functional() = true` with a real device
(GeForce GTX 1080 Ti), a real `:gpu` solve completes (205s), and the node
can push to the repo via the same mechanism live deployment already uses
(`GITHUB_TOKEN` over HTTPS -- Documenter's own `deployconfig.jl` never
picks SSH here, since `DOCUMENTER_KEY` isn't passed by that caller). A
full docs build there took 2265s.

Design: *publish then upgrade*, not probe-then-fallback. The existing
`build` job is untouched and always runs first -- so docs publishing
never depends on a self-hosted box being online, reachable, or fast. A
new `build-gpu` job runs only once `build` has already deployed
(`needs: build`) and, with `continue-on-error: true`, can never fail the
workflow: on success it redeploys the same site with real GPU numbers;
on failure, or if the runner is queued behind other work and the
`gpu_timeout_minutes` budget runs out, the site `build` already published
is simply left as is.

New inputs, both optional and both default to reproducing today's
behaviour exactly for every existing caller:

- `gpu_runner` (default `''`): JSON array of self-hosted runner labels,
  e.g. `'["occidata"]'`. Empty disables `build-gpu` entirely -- it shows
  as `skipped` in the run graph, at no cost to callers that never set it.
- `gpu_timeout_minutes` (default `120`): budget for a *cold* self-hosted
  build. Self-hosted depots persist across runs and get purged by
  maintenance jobs (occidata's Monday 02:30 UTC cache purge), so the
  first build after one is the number to budget for, not a warm rebuild.

`build-gpu` also carries the self-hosted adaptations `ci.yml` already
established and this file's own `build` job (GitHub-hosted) has no need
of:

- resolve the real Julia depot path instead of assuming `~/.julia`
  (`setup-julia` does not set `JULIA_DEPOT_PATH`, and self-hosted depots
  vary by machine);
- redirect `TMPDIR` off the node's periodically-wiped `/tmp` before any
  Julia step runs -- a docs build is longer than a test run, so it will
  almost certainly cross a wipe boundary;
- purge-and-retry the compiled cache once on failure, since a persistent
  depot can drift out of sync with a different branch's Manifest.

`build-gpu` only runs for `push`/`tag` triggers, never for
`pull_request` -- PR-preview docs builds stay GitHub-hosted-only and
fast, matching what #885 itself proposed.

No change to the `build` job. Every existing caller (all packages using
this reusable workflow) is unaffected: `gpu_runner` defaults to `''`, so
`build-gpu`'s `if` is false and it shows `skipped`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant