Skip to content

fix(ci): give each checkout its own bazel output base in container-run.sh - #11441

Open
basvandijk wants to merge 2 commits into
masterfrom
bas/container-run-per-checkout-output-base
Open

fix(ci): give each checkout its own bazel output base in container-run.sh#11441
basvandijk wants to merge 2 commits into
masterfrom
bas/container-run-per-checkout-output-base

Conversation

@basvandijk

Copy link
Copy Markdown
Collaborator

Problem

Running ./ci/container/container-run.sh from two checkouts (e.g. a git worktree next to the main checkout) and invoking bazel in both kills the first bazel server:

Server terminated abruptly (error code: 14, error message: 'recvmsg:Connection reset by peer', log file: '/home/ubuntu/.cache/bazel/_bazel_ubuntu/6d065581cce7ad9076e3b8db2b3afaf0/server/jvm.out')

Every checkout is mounted at /ic and shares ~/.cache, so all containers use the same output base (6d065581... is md5("/ic")). Each container has its own PID namespace, and Bazel's client only recognizes a live server by its pid and /proc/<pid>/stat start time, so the client in the second container treats the first container's server as dead (restart_reason=pid_file_but_no_server), starts a second server in the same output base, and its launcher rewrites server.pid.txt. The first server's PidFileWatcher then halts it. Because the containers use --network=host, the second client can even end up connected to the first container's server and run its command against the other checkout (this happens when both servers get the same pid number in their namespaces). The same collision occurs between a VS Code devcontainer and a container-run.sh container on different checkouts.

Changes

  • container-run.sh gives every checkout its own bazel output base, ~/.cache/bazel/_bazel_ubuntu/<basename>-<sha256 of the host path>, by pointing BAZELRC at a generated one-line rc file under <cache dir>/container-run/. Bazel reads $BAZELRC in addition to the workspace .bazelrc (including its user.bazelrc import) and ~/.bazelrc, so the rest of the configuration is unchanged; the install base, the repository cache and the repo contents cache remain shared. The variable only exists inside the container, so host-side bazel is unaffected.
  • Linked git worktrees are supported: the main repository's .git directory is bind-mounted at its host path so that git works inside the container (--config=stamped, ci/scripts/rust-lint.sh, ic-admin's build script under cargo). gc's automatic worktree pruning is disabled in the container (gc.worktreePruneExpire=never) because linked worktrees look prunable from inside it.
  • README: replaced the "How to run parallel bazel tests" section.

Notes

  • The first run from a checkout after this change does a cold build (the remote cache mitigates). The old shared output base ~/.cache/bazel/_bazel_ubuntu/6d065581cce7ad9076e3b8db2b3afaf0 can be deleted afterwards.
  • Two containers started from the same checkout still share an output base; use podman exec for a second shell.
  • .devcontainer/devcontainer.json is unchanged and keeps the default output base. It no longer collides with container-run.sh containers, but two devcontainers on different checkouts still would; that can be a follow-up.

Testing

Verified on a devenv with an isolated cache dir (-c) from the main checkout and from a linked worktree: each container got its own output base; a server started from the main checkout survived a second container from the worktree starting its own server, and its server_pid was unchanged afterwards; inside the worktree container git rev-parse HEAD, git status, bazel/workspace_status.sh --stamp and git config gc.worktreePruneExpire (never) all worked. bazel run //pre-commit:shfmt-check passes.

🤖 Generated with Claude Code

…n.sh

Containers started from different checkouts (git worktrees or clones)
all mount the checkout at /ic and share ~/.cache, so they used a single
bazel output base (md5("/ic")). Bazel's client only recognizes a live
server by its pid and /proc start time, which it cannot see across the
containers' PID namespaces, so the second container started another
server in the same output base and the first server halted itself:

    Server terminated abruptly (error code: 14, error message:
    'recvmsg:Connection reset by peer', ...)

With --network=host the second client could even be served by the first
container's server, i.e. run against the other checkout.

container-run.sh now points $BAZELRC at a generated rc file that sets
--output_base to ~/.cache/bazel/_bazel_ubuntu/<basename>-<sha of the
host path>, keeping the install base and the repository caches shared
and leaving the rest of the bazel configuration (.bazelrc, user.bazelrc)
untouched. For linked git worktrees it also bind-mounts the main
repository's .git directory at its host path so that git works inside
the container, and disables gc's automatic worktree pruning there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unsanitized or excessively long checkout names can break every Bazel invocation or prevent container startup.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Isolates Bazel output bases per checkout and adds linked Git worktree support.

Changes:

  • Generates a checkout-specific Bazel rc file.
  • Mounts linked worktrees’ common Git directory.
  • Documents concurrent checkout behavior and limitations.
File summaries
File Description
ci/container/README.md Documents parallel checkout behavior and cache usage.
ci/container/container-run.sh Configures isolated output bases and worktree mounts. The checkout basename must be sanitized and bounded before use in rc paths and filenames.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread ci/container/container-run.sh Outdated
… key

The basename is copied into the generated `startup --output_base=...`
rc line, into the rc file name passed via the comma-separated $BAZELRC
variable, and into a directory name, so whitespace, quotes, backslashes
or commas in a checkout name would break bazel in that container. Keep
only [A-Za-z0-9._-], cap it at 64 characters, and rely on the path hash
for uniqueness. Keys of ordinary checkout names are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@basvandijk
basvandijk requested a balanced review from Copilot September 3, 2026 19:51
@basvandijk
basvandijk marked this pull request as ready for review September 3, 2026 19:52
@basvandijk
basvandijk requested a review from a team as a code owner September 3, 2026 19:52
@github-actions github-actions Bot added the @infra label Sep 3, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Both reviewed files have no unresolved issues, and the changes were verified for concurrent checkouts and linked worktrees.

Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new
  • Review effort level: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants