[fix](ci) add swap space in deploy workflows to avoid runner OOM kills - #4030
Merged
Merged
Conversation
The cron/manual deploy jobs have been failing intermittently for days with "The runner has received a shutdown signal" during the zh-CN docusaurus build. The same commit passes and fails alternately, and no JS heap OOM is ever reported, which points to the runner VM itself being killed by system-level memory exhaustion: the main node process is allowed an 8G heap and webpack/terser worker processes push the total past the 16G physical memory of the standard runner. Add an extra 8G swapfile (on the /mnt data disk, alongside the built-in 4G one) before the build so memory spikes spill to swap instead of getting the VM killed. Also print free/swapon output for easier debugging of future failures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & Why
The
Cron DeployandManual Deploy And Upload To OSSjobs have been failing intermittently for the past several days (e.g. 30743875275, 30751563874), and went into a consecutive-failure streak on Aug 2.Diagnosis from the run logs:
##[error]The runner has received a shutdown signalin the middle of the Build step — the runner VM itself is killed, not a build error and not a workflow timeout.3862366c) both passed and failed repeatedly across Aug 1–2, and the runner image version is identical between passing and failing runs (ubuntu-24.04 / 20260720.247.2), ruling out code or image changes.JavaScript heap out of memoryerror — the classic signature of system-level memory exhaustion killing the runner service: the main node process is allowed--max-old-space-size=8192and the webpack/terser worker processes push the total past the standard runner's 16G physical memory.Fix
Add an extra 8G swapfile before the Build step in both deploy workflows, so memory spikes spill into swap instead of getting the VM OOM-killed:
/mntdata disk (~65G free) instead of the root disk, which the workspace and build output already fill; namedswapfile2because the runner image ships a built-in 4G/mnt/swapfile— total 12G swap after this change.free -h/swapon --showoutput is printed so future failures can be triaged against the memory baseline directly from the logs.This is a mitigation for the flaky OOM kills. A follow-up improvement worth considering is splitting
docusaurus build --locale en --locale zh-CNinto two separate invocations so each locale's peak memory is isolated in its own process.🤖 Generated with Claude Code