Skip to content

[fix](ci) add swap space in deploy workflows to avoid runner OOM kills - #4030

Merged
morningman merged 1 commit into
apache:masterfrom
morningman:add-swap-to-deploy-workflows
Aug 3, 2026
Merged

[fix](ci) add swap space in deploy workflows to avoid runner OOM kills#4030
morningman merged 1 commit into
apache:masterfrom
morningman:add-swap-to-deploy-workflows

Conversation

@morningman

Copy link
Copy Markdown
Contributor

What & Why

The Cron Deploy and Manual Deploy And Upload To OSS jobs have been failing intermittently for the past several days (e.g. 30743875275, 30751563874), and went into a consecutive-failure streak on Aug 2.

Diagnosis from the run logs:

  • Every failure ends with ##[error]The runner has received a shutdown signal in the middle of the Build step — the runner VM itself is killed, not a build error and not a workflow timeout.
  • The same commit (3862366c) both passed and failed repeatedly across Aug 1–2, and the runner image version is identical between passing and failing runs (ubuntu-24.04 / 20260720.247.2), ruling out code or image changes.
  • Failures consistently happen during the memory-heaviest phase (en static generation finished, zh-CN webpack build in progress), after several minutes of log silence, and with no JavaScript heap out of memory error — the classic signature of system-level memory exhaustion killing the runner service: the main node process is allowed --max-old-space-size=8192 and the webpack/terser worker processes push the total past the standard runner's 16G physical memory.

Fix

Add an extra 8G swapfile before the Build step in both deploy workflows, so memory spikes spill into swap instead of getting the VM OOM-killed:

  • The swapfile is placed on the /mnt data disk (~65G free) instead of the root disk, which the workspace and build output already fill; named swapfile2 because the runner image ships a built-in 4G /mnt/swapfile — total 12G swap after this change.
  • free -h / swapon --show output is printed so future failures can be triaged against the memory baseline directly from the logs.

This is a mitigation for the flaky OOM kills. A follow-up improvement worth considering is splitting docusaurus build --locale en --locale zh-CN into two separate invocations so each locale's peak memory is isolated in its own process.

🤖 Generated with Claude Code

The cron/manual deploy jobs have been failing intermittently for days
with "The runner has received a shutdown signal" during the zh-CN
docusaurus build. The same commit passes and fails alternately, and no
JS heap OOM is ever reported, which points to the runner VM itself being
killed by system-level memory exhaustion: the main node process is
allowed an 8G heap and webpack/terser worker processes push the total
past the 16G physical memory of the standard runner.

Add an extra 8G swapfile (on the /mnt data disk, alongside the built-in
4G one) before the build so memory spikes spill to swap instead of
getting the VM killed. Also print free/swapon output for easier
debugging of future failures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@morningman
morningman merged commit c80c54d into apache:master Aug 3, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant