What the CI spends per push, and why fork pull requests run zero jobs

[torial]

We surveyed vlang/v's Actions workflows from the API on 28 October (authenticated gh), replayed the trigger filters of all 64 workflow files at 5cde58423, and timed pieces of the build and the test suite on a 4-vCPU VM. This post has the four findings with how sure we are of each, two small pull requests, and the full map at the end. Nothing here proposes a rewrite of the CI; the numbers are offered for the maintainers to decide on.

1. Fork pull requests run zero jobs because their approval expires

The core workflows have had no successful run since August (CI Linux last green 14 Aug, Windows gcc 28 Aug, the sanitizers 16 Aug, tools 27 Aug, other-CI 26 Aug). Of the last 100 runs of each, about 71 were still queued, 23 cancelled, 5 awaiting approval, and none succeeded.

Part of that is a state the CLI misreports. gh run view on a failed run from a fork says "This run likely failed because of a workflow file issue", but the run page's annotation reads:

This workflow run required approval but was not approved before it expired.

Every zero-job failure we read carries it: the 20 workflows on PR #31306's last push, the 39 on PR #31358's, and 93 of the last 100 failed pull-request runs on 29 October, all from forks. A fork pull request's workflows wait for a maintainer's approval on each push and fail with zero jobs when it lapses; PR #31161's first push ran 42 jobs, its next push one, and the two after it none. The workflow files are not at fault (actionlint 1.7.7 reports nothing for linux_ci.yml, and the fork's .github/workflows tree was identical to master's at the merge). The lever is the repository's fork-PR approval setting under Actions, which we cannot read.

Sure of: the annotation text and the counts above. Not checked: what the setting is currently set to.

2. The unit-test matrix is the largest single cost

compiler_ci.yml runs ten shards of v run vlib/v/test_all.vsh --units-only --shard N/10 on every push to master. One shard (--shard 1/10) did not finish in 40 minutes on the 4-vCPU VM (17 files done, vlib/v/compiler_tests/ files at 10 to 143 s each); the workflow allows 120. On that evidence ten shards cost at least 400 job-minutes per master push, against 333 job-minutes for the last green run of every other workflow together, and master takes about 28 pushes a day. This is our VM, not a GitHub runner, so it is a floor. A nightly run, plus a pull-request run of what a change touches, would remove most of it.

3. Most jobs rebuild V

Of 198 job instances in 126 job templates, 132 run make (or makev.bat) and 8 use the setup-v action. A cold make took 167 s (median of 3) and 1.78 GB RSS on the VM; the survey's "Build V" steps run 1.5 to 5.8 minutes. 83 of the 132 are repeats inside one workflow, an upper bound on what sharing a build could remove (matrix entries on other operating systems cannot share a Linux binary).

Two pull requests:

• ci: skip the disabled package2-linux job instead of starting 3 runners #31418 moves the if: ${{ false }} of the disabled package2-linux job from its step to the job, so three runners are no longer started to skip one step.
• A second one, to follow, builds V once in docs_ci.yml and hands v and thirdparty/tcc to its four jobs as a one-day artifact; it is being run on a fork first so the artifact round trip has a measured time on real runners before it is offered.

4. What fires per change

Replaying each workflow's push and pull_request filters (branch and path rules, including ! negations) against six example changes, with the survey's minutes split across a workflow's jobs by instance count:

Change	Event	Job instances today	Measured job-min today (unmeasured instances)	Instances if PR and path-triggered jobs only
compiler change	pull_request	131	307 (77)	41
library change	pull_request	109	273 (69)	37
tool change	pull_request	46	85 (38)	15
example only	pull_request	59	183 (33)	18
docs only	pull_request	10	28 (5)	7
workflow only	pull_request	6	0 (6)	5

The classification behind the last column (every pull request / path-triggered / merge to master only / nightly / manual) is a proposal; the map below marks it per workflow, with a star where it is a judgement call.

The suite, briefly

v test runs min(CPUs, a memory budget, 4) workers; CI Linux sets VJOBS=4 on a 4-CPU runner, so it is CPU-bound there. In the 760 of 6,166 test files we timed (the first directories alphabetically, so not a random sample), 74% of the time is compiling, not running: 31 minutes of compile to 11 of run. The largest single files: vlib/v/markused/custom_str_test.v (72 s compile, 156 s run), vlib/mcp/mcp_test.v (70 + 69 s), and several in db/redis, db/pg and encoding/cose at 20 to 37 s of compile for a run of under a second. A warm module cache makes a net.http program build 15 times faster than cold (44 s to 2.9 s on the VM); every CI job starts cold.

Not verified

• Nothing in sections 2 to 4 ran on GitHub runners. All timings are from one 4-vCPU VM that restarted several times; medians of 3 for builds, single runs for the test timings.
• The full v test vlib did not complete; the sample is 760 files.
• Job-minutes per job inside a workflow are the survey's workflow total split by instance count.

The map

One row per workflow file at 5cde58423. Jobs is templates / instances after matrix expansion. Timeout in minutes. Minutes are the survey's job-minutes for the last successful run of that workflow ("unmeasured" when it had no timed run in the last month). Class is the proposal of section 4; a star marks a judgement call.

64 workflows

Workflow	Triggers	Jobs	Timeout	Gets V	Minutes	Class
add-voting-info.yml	issues	1 / 1	60	none	unmeasured	MANUAL
benchmark_footprint_json_decode.yml	push [master] paths(2)	1 / 1	5	make	unmeasured	PATH
benchmark_footprint_json_encode.yml	push [master] paths(4)	1 / 1	5	make	unmeasured	PATH
bootstrapping_ci.yml	workflow_dispatch, push [master] ignore(5+3 negated), pull_request ignore(5+3 negated)	1 / 2	20	make	34.2	MASTER*
build_on_ubuntu_25_ci.yml	workflow_dispatch, push [master] ignore(3+2 negated), pull_request ignore(3+2 negated)	1 / 1	30	make	2.2	MASTER*
c2v_ci.yml	push [master] ignore(6+2 negated), pull_request ignore(6+2 negated)	2 / 2	20-30	make	9.8	NIGHTLY*
compiler_ci.yml	workflow_dispatch, push [master] paths(1), pull_request paths(1)	2 / 11	120	make	10.4	NIGHTLY* PATH
cross_ci.yml	push [master] ignore(3+2 negated), pull_request ignore(3+2 negated)	3 / 3	20	make	22.6	MASTER*
db_ci.yml	workflow_dispatch, push [master] ignore(4+1 negated), pull_request ignore(4+1 negated)	3 / 3	30	make	unmeasured	PATH
debug.yml	workflow_dispatch	1 / 1	121	make	unmeasured	MANUAL
docker_ci.yml	push [master] ignore(3+2 negated), pull_request ignore(3+2 negated)	3 / 3	150-241	make	unmeasured	NIGHTLY*
docs_ci.yml	push [master] ignore(2+1 negated), pull_request ignore(2+1 negated)	4 / 4	5-20	make	24.4	NIGHTLY* PATH
docs_site_ci.yml	workflow_dispatch, schedule, push [master] paths(3), pull_request paths(6), schedule	1 / 1	20	make	3.3	PATH
fast_pages.yml	workflow_dispatch, schedule, schedule	1 / 1	60	none	unmeasured	MANUAL
freebsd_ci.yml	workflow_dispatch, push [master] ignore(3+3 negated), pull_request paths(49)	3 / 3	20-30	none	unmeasured	MASTER*
gen_vc_ci.yml	workflow_dispatch, pull_request ignore(7+3 negated), push [master] ignore(7+3 negated)	2 / 2	20	make	4.3	MASTER PATH
gg_regressions_ci.yml	push [master] paths(6), pull_request paths(6)	1 / 1	20	make	unmeasured	PATH
h2spec.yml	push paths(6), pull_request paths(6)	1 / 1	60	make	unmeasured	PATH
hub_docker_ci.yml	workflow_dispatch, push	2 / 5	20-60	none	unmeasured	MANUAL
linux_ci.yml	workflow_dispatch, push [master] ignore(3+2 negated), pull_request ignore(3+2 negated)	4 / 6	121	make	unmeasured	MASTER* OFF PR
macos_ci.yml	workflow_dispatch, push [master] ignore(3+2 negated), pull_request ignore(3+2 negated)	2 / 9	90-120	make	24.4	MASTER*
macos_gitly_ci.yml	push [master], pull_request, workflow_dispatch	1 / 1	120	make	unmeasured	OFF
makev_ci.yml	push [master] paths(20), pull_request paths(20), workflow_dispatch	1 / 2	5	none	0.8	PATH
module_docs_ci.yml	workflow_dispatch, pull_request paths(8), push [master] paths(8)	1 / 1	20	make	unmeasured	PATH
module_docs_lint.yml	push [master] paths(1), pull_request paths(1)	1 / 1	60	none	unmeasured	PATH
more_extensive_but_slower_tests_ci.yml	push [master] ignore(4+1 negated), pull_request ignore(4+1 negated)	1 / 3	20	make	27.8	NIGHTLY*
multiwindow_musl_exclusion_smoke.yml	workflow_dispatch, push [smoke/x-multiwindow-windows-20260717] paths(8)	1 / 1	20	make	unmeasured	PATH
notices_ci.yml	push [master] paths(5), pull_request paths(5), schedule, workflow_dispatch, schedule	2 / 4	60-180	make	unmeasured	NIGHTLY PATH
openbsd_ci.yml	workflow_dispatch, push [master] paths(8), pull_request paths(48)	2 / 2	30	none	unmeasured	MASTER*
other_ci.yml	workflow_dispatch, push [master] ignore(3+2 negated), pull_request ignore(3+2 negated)	6 / 6	20-80	make	unmeasured	MASTER* NIGHTLY* PATH PATH* PR
paths_ci.yml	workflow_dispatch, push [master] ignore(3+2 negated), pull_request ignore(3+2 negated)	3 / 3	20-30	make	unmeasured	MASTER*
periodic_ci.yml	workflow_dispatch, schedule, push [master] paths(1), pull_request paths(1), schedule	1 / 3	30	make	unmeasured	NIGHTLY
prebuilt.yml	workflow_dispatch, release	4 / 4	5	none	unmeasured	MANUAL
process_regressions_ci.yml	pull_request paths(5), push [master] paths(5), workflow_dispatch	1 / 3	30	none	unmeasured	PATH
puzzle_vibes_ci.yml	workflow_dispatch, push [master] paths(8), pull_request paths(8)	1 / 1	20	make	2.8	PATH
release_ci.yml	workflow_dispatch, pull_request paths(1), push paths(1)	2 / 11	60	make	unmeasured	MANUAL
resolve-pr-28232.yml	push [fix-master-ci-batch-2-current]	1 / 1	60	none	unmeasured	OFF
riscv64_linux_ci.yml	workflow_dispatch, push [master] ignore(3+2 negated), pull_request paths(26)	1 / 1	120	none	unmeasured	NIGHTLY*
s390x_linux_ci.yml	workflow_dispatch, push [master] ignore(3+2 negated), pull_request paths(26)	1 / 1	120	none	unmeasured	NIGHTLY*
sanitized_ci.yml	workflow_dispatch, push [master] paths(26), pull_request paths(24)	2 / 8	30	make+setup-v	unmeasured	MASTER* NIGHTLY*
sdl_ci.yml	workflow_dispatch, push [master] paths(6), pull_request paths(6)	1 / 1	20	make	3.8	PATH
symlink_ci.yml	workflow_dispatch, push [master] paths(2), pull_request paths(2)	2 / 4	60	make	8.8	PATH
tccbin_automation_contract.yml	pull_request, merge_group	4 / 4	5-45	make	unmeasured	PATH*
tccbin_issue_reconcile.yml	workflow_dispatch, schedule, schedule	1 / 1	30	make	unmeasured	MANUAL
tccbin_revalidate.yml	workflow_dispatch, schedule, schedule	2 / 2	10-30	make	unmeasured	MANUAL
tccbin_source_recovery.yml	workflow_dispatch, schedule, workflow_run, schedule	4 / 4	5-30	make	unmeasured	MANUAL
termux_ci.yml	workflow_dispatch, push [master] ignore(3+2 negated), pull_request paths(24)	2 / 2	45	make	6.6	MASTER*
test_selection_ci.yml	push [master] paths(7), pull_request paths(7), workflow_dispatch	1 / 3	20	none	15.3	PATH
time_ci.yml	push [master] paths(4), pull_request paths(4)	3 / 3	60	make	24.5	PATH
toml_ci.yml	push [master] paths(4), pull_request paths(4)	1 / 1	20	setup-v	4.8	PATH
tools_ci.yml	push [master] paths(6), pull_request paths(6)	7 / 14	20-40	make	unmeasured	MASTER* NIGHTLY* PATH
update_tccbin.yml	workflow_dispatch, schedule, schedule	5 / 5	30-120	make	unmeasured	MANUAL
v_apps_and_modules_compile_ci.yml	push [master] ignore(4+1 negated), pull_request ignore(4+1 negated), schedule, schedule	3 / 5	20-60	make	unmeasured	NIGHTLY*
vab_ci.yml	workflow_call, push [master] paths(5), pull_request paths(5)	2 / 2	20	make	12.4	PATH
vinix_ci.yml	workflow_call, pull_request ignore(3+1 negated), push [master] ignore(3+1 negated)	1 / 1	20	make	unmeasured	MASTER*
vpm_ci.yml	push [master] paths(2), pull_request paths(2)	2 / 10	60	make	unmeasured	PATH
vsl_and_vtl_compile_ci.yml	push [master] ignore(4+1 negated), pull_request ignore(4+1 negated)	2 / 4	25-30	make	unmeasured	NIGHTLY*
vup_works.yml	push [master] ignore(5+2 negated), pull_request ignore(5+2 negated)	1 / 3	45	make	57.5	MASTER*
wasm_backend_ci.yml	push [master] paths(19), pull_request paths(19)	1 / 2	20	make	21.5	PATH
websockets_ci.yml	push [master] ignore(3+2 negated), pull_request ignore(3+2 negated)	1 / 1	121	make	10.2	PATH
windows_ci_gcc.yml	push [master,smoke/**] ignore(3+3 negated), pull_request ignore(3+4 negated)	2 / 2	150	make	unmeasured	MASTER* PATH
windows_ci_msvc.yml	push [master,smoke/**] ignore(3+3 negated), pull_request ignore(3+4 negated)	2 / 2	70-90	make	unmeasured	MASTER* PATH
windows_ci_tcc.yml	workflow_dispatch, push [master] ignore(3+3 negated), pull_request ignore(3+4 negated)	2 / 2	60	make	unmeasured	PATH PR
workflow_lint.yml	push [master] paths(1), pull_request paths(1)	1 / 1	60	none	unmeasured	PATH

The full page with the per-file timings, the build-once design and the patch bytes is available on request.

[torial]

Patch 2 is open as #31424. It was measured first on GitHub's runners as two fork pull requests on the same base (77e3480757): the patched docs_ci.yml and an unpatched control, each run once.

job	control	patched
build V	inside each of the four jobs	once, 2m04s
artifact round trip	-	3 s to download, under 1 s to unpack
check-markdown	15m27s	12m43s
report-missing-dots	2m24s	14 s
report-missing-fn-doc	1m42s	18 s
report-wrong-examples	4m18s	2m52s
job time, all jobs	23m51s	18m11s

All four jobs were green off the copied binary. One observation separate from the patch: on the first run check-markdown's step took 19m44s on a slow runner and the job was cancelled at its 20-minute timeout-minutes. Upstream's last two green runs had that step at 12m36s and 16m13s inside a 19-minute job, so the timeout is already tight whether or not the build is shared.

On running a workflow's tests from one script instead of one job each (a suggestion from JalonSolov, relayed to us after this was posted): it is the strongest form of the lever in section 3, and the map gives its bound. 83 of the 132 build-V instances are repeats inside one workflow. One job per workflow (per operating system, since a matrix entry on another OS cannot share the binary) removes those 83 builds and the 83 runner starts and queue waits that go with them, and under a backed-up queue the waits are the part that hurts most. #31424 is the artifact form of the same change for one workflow; the script form avoids even the artifact round trip, which at 3 s is not where the time goes.

The trade: the jobs then run in sequence on one runner, so a workflow's wall time becomes the sum of its jobs less the repeated builds, and one named red job in the checks list becomes one red whose step has to name the failing suite. Where the map says it would pay, counting instances that each build V (some of these are spread over operating systems, so each workflow's own figure is at most this): tools_ci (14), vpm_ci (10), other_ci (6), symlink_ci (4), and docs_ci (4, which #31424 covers). compiler_ci's ten shards (11 instances) are sharded for parallelism; a script there would turn a 40-minute-plus shard into a serial run of all ten, so that one wants to stay as separate runners or move to a nightly, as section 2 proposes.

The script form has not been measured on runners from here; the figures above are the map's counts.