Skip to content

builder-agent: use a pre-mounted /var/lib/buildkit and bound BuildKit GC - #339

Merged
rgarcia merged 3 commits into
mainfrom
hypeship/builder-agent-premounted-root
Aug 4, 2026
Merged

builder-agent: use a pre-mounted /var/lib/buildkit and bound BuildKit GC#339
rgarcia merged 3 commits into
mainfrom
hypeship/builder-agent-premounted-root

Conversation

@rgarcia

@rgarcia rgarcia commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Extract ensureBuildkitRoot: when the host has already attached a disk at
/var/lib/buildkit the agent uses it directly; otherwise it mounts the same
3 GB tmpfs as before with identical command and error behavior. Mountpoint
detection reads /proc/self/mounts, which reflects the guest's own mount
namespace. If persistent-cache GC bounds are configured, a missing mount
fails fast instead of applying large-disk thresholds to the tmpfs fallback.

Add optional GC bounds to the guest BuildConfig (unused by the host for
now). When both bounds are set, buildkitd.toml gains an OCI worker gcpolicy
with gc=true and quoted human-readable reservedSpace/maxUsedSpace sizes.
Bounds must both be at least 1 MiB, and reservedSpace must be less than
maxUsedSpace. The deprecated gckeepstorage is not emitted: it only maps to
reservedSpace, so nothing would reclaim above the floor and a fixed-size
disk could fill to ENOSPC. Sizes are quoted because BuildKit decodes them
with units.RAMInBytes; a bare integer would be read as bytes.

Tests cover the mount decision (mounted, unmounted, check and mount
failures), /proc/self/mounts parsing, a TOML golden for the worker
section, and a decode regression proving the emitted sizes parse back to
the intended byte counts via docker/go-units (the same decoder BuildKit
uses).


Stack created with GitHub Stacks CLIGive Feedback 💬


Note

Medium Risk
Changes how BuildKit stores cache data and when builds fail (persistent mount required with GC bounds); host wiring is not in this diff, but misconfigured bounds or shared volumes could cause failed builds or ENOSPC without GC.

Overview
The builder agent no longer always mounts a 3 GB tmpfs at /var/lib/buildkit. ensureBuildkitRoot reuses an existing mount (e.g. host-attached persistent disk) when /proc/self/mounts shows one; otherwise it keeps the same tmpfs behavior. If GC byte bounds are set but the path is not mounted, the build fails fast instead of applying disk-sized GC to tmpfs.

CacheGCReservedBytes and CacheGCMaxUsedBytes are added on guest and host BuildConfig (host still leaves them zero). When both are set, buildkitd.toml gets an OCI worker gcpolicy with quoted reservedSpace / maxUsedSpace (BuildKit units.RAMInBytes decoding); validation enforces ≥1 MiB and reserved < max, and gckeepstorage is omitted on purpose.

Tests cover mount decisions, mount table parsing, GC TOML output, and size decode regression. docker/go-units is a direct dependency.

Reviewed by Cursor Bugbot for commit 02706f7. Bugbot is set up for automated code reviews on this repo. Configure here.

Extract ensureBuildkitRoot: when the host has already attached a disk at
/var/lib/buildkit the agent uses it directly; otherwise it mounts the same
3 GB tmpfs as before with identical command and error behavior. Mountpoint
detection reads /proc/self/mounts, which reflects the guest's own mount
namespace.

Add optional GC bounds to the guest BuildConfig (unused by the host for
now). When both bounds are set, buildkitd.toml gains an OCI worker gcpolicy
with gc=true and quoted human-readable reservedSpace/maxUsedSpace sizes.
The deprecated gckeepstorage is not emitted: it only maps to reservedSpace,
so nothing would reclaim above the floor and a fixed-size disk could fill
to ENOSPC. Sizes are quoted because BuildKit decodes them with
units.RAMInBytes; a bare integer would be read as bytes.

Tests cover the mount decision (mounted, unmounted, check and mount
failures), /proc/self/mounts parsing, a TOML golden for the worker
section, and a decode regression proving the emitted sizes parse back to
the intended byte counts via docker/go-units (the same decoder BuildKit
uses).
@rgarcia
rgarcia marked this pull request as ready for review August 3, 2026 23:44
@hiroTamada
hiroTamada self-requested a review August 4, 2026 15:42

@hiroTamada hiroTamada left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved — the mount detection and tests look solid. two non-blocking edge cases worth considering before the host enables persistent caches:

  • lib/builds/builder_agent/main.go:886-889 — if GC bounds are configured but the persistent mount is missing, this silently falls back to a 3 GB tmpfs with limits intended for a larger disk, which can reach ENOSPC before GC starts. consider failing fast when nonzero bounds imply that the mount must exist.
  • lib/builds/builder_agent/main.go:769-775 — consider rejecting reservedBytes >= maxUsedBytes and positive values below 1 MiB; the current conversion can emit a nonsensical policy, including "0MB", from positive inputs.

@sjmiller609 sjmiller609 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

builder_agent/main.go: crosses 1k lines by end of stack, optionally extract some code to buildkitroot.go.

Otherwise, no objections, looks good.

@rgarcia

rgarcia commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

@hiroTamada addressed both edge cases in 69a4179: configured GC bounds now require an existing /var/lib/buildkit mount instead of falling back to tmpfs, and malformed bounds are rejected (both must be at least 1 MiB; reserved must be below max). Added focused mount and validation regressions; go test -race ./lib/builds/builder_agent -count=3 passes.

@rgarcia

rgarcia commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

@sjmiller609 took the optional split in 02706f7: BuildKit root setup and mount detection now live in buildkitroot.go. The upper Builder integration keeps its version-stamp/reset behavior in that file as well, reducing main.go by the root-lifecycle block and keeping that behavior isolated. Focused tests, race tests, vet, and build pass.

@rgarcia
rgarcia merged commit 652f94e into main Aug 4, 2026
11 of 12 checks passed
@rgarcia
rgarcia deleted the hypeship/builder-agent-premounted-root branch August 4, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants