Promote synvya-staging to synvya: low-mem EC2 build optimizations#45
Merged
alejandro-runner merged 5 commits intosynvyafrom Apr 29, 2026
Merged
Promote synvya-staging to synvya: low-mem EC2 build optimizations#45alejandro-runner merged 5 commits intosynvyafrom
alejandro-runner merged 5 commits intosynvyafrom
Conversation
The Synvya EC2 instances run Rust release builds during deploy (docker compose build) and QA (cargo test). On t3.medium (4 GB RAM, 2 vCPU) these can OOM, hanging at random crates such as rand v0.8.5. Swap on the host plus a cargo jobs limit reliably prevents the OOM, but until now had to be set up manually. - Add scripts/ec2-prepare-host.sh: idempotent helper that ensures a 4 GB /swapfile is active and persisted via /etc/fstab, and writes ~/.cargo/config.toml with [build] jobs=2 if missing. - Call it from deploy-staging, deploy-production, qa-staging, and qa-production before the build/test step. - Pass --build-arg CARGO_BUILD_JOBS=2 to docker compose build so the in-container cargo build is also constrained. - Dockerfile: accept CARGO_BUILD_JOBS as ARG and forward to ENV; defaults to empty (no change for contributors with bigger RAM). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two changes that make the warm path of Synvya deploys actually warm: 1. Dockerfile: split the rust-builder stage into a deps-only build followed by a real-source build. The first cargo invocation runs against stub src files for every workspace member, compiling just the ~400 transitive dependencies and caching them as a Docker layer. The second invocation overwrites the stubs with real code; cargo only recompiles workspace crates, reusing dep .rlibs. Net effect: code-only changes no longer trigger a full dependency recompile on every deploy. 2. Workflow: drop the `-a` flag from `docker builder prune` and `docker image prune`. The `-a` variant deletes referenced build cache and tagged images, guaranteeing a cold build every time. The non-`-a` variants still clean up dangling images and unreferenced build cache for disk hygiene. Combined with the earlier swap + cargo -j 2 commit, this should let the t3.medium handle warm builds quickly and survive cold builds without OOMing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
ci(ec2): swap + cargo -j 2 + Docker dep cache for low-mem deploys
…che mounts The previous attempt at a deps-cache stage (stub every workspace member's source, build deps with stubs, then COPY real source) fails at compile time because the Keycast binary's dependency graph crosses workspace boundaries: keycast → keycast_api → keycast_core::types, etc. Stubbing keycast_core's lib.rs to empty makes keycast_api fail to compile inside Stage A. The stub pattern only works for projects with no workspace-internal deps. Keycast has them everywhere. Fix: collapse to a single rust-builder stage and use BuildKit cache mounts for /usr/local/cargo/registry, /usr/local/cargo/git, and /app/target. The mounts persist on the host's BuildKit instance across builds, so unchanged dep crates stay compiled without any source-stubbing trickery. Built artifacts are copied out of target/ to /artifacts/ before the RUN ends, since cache mounts are not part of the resulting image layer; the runtime stage now COPYs from /artifacts/. Workflow exports DOCKER_BUILDKIT=1 and COMPOSE_DOCKER_CLI_BUILD=1 to be explicit (compose v2 enables BuildKit by default). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
fix(docker): use BuildKit cache mounts for cargo deps
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Promotes the build-time memory optimizations validated on staging to production.
What's in this merge
ci(ec2): swap + cargo -j 2 + Docker dep cache for low-mem deploysscripts/ec2-prepare-host.sh: idempotent swap (4 GB) +~/.cargo/config.tomlwith[build] jobs = 2--build-arg CARGO_BUILD_JOBS=2-afromdocker builder/image pruneso referenced layers and images survive between deploysfix(docker): use BuildKit cache mounts for cargo depsRUN --mount=type=cachefor/usr/local/cargo/registry,/usr/local/cargo/git,/app/target/artifacts/before the cache mount disappears; runtime stage COPYs from thereDOCKER_BUILDKIT=1andCOMPOSE_DOCKER_CLI_BUILD=1Validated on staging
Staging deploy succeeded after these changes — t3.medium no longer hangs at
rand v0.8.5or other crates during the cargo release build.Self-contained on production EC2
The workflow, prep script, and Dockerfile are all merged together. On the first production deploy:
git pullbrings the new files onto the prod EC2.ec2-prepare-host.shcreates the swapfile + cargo jobs config before any build.-j 2— slower but won't OOM.No manual prep on the production host required.
Test plan