Skip to content

revert(image): go back to glibc until musl can link inside the builder - #331

Merged
BryanFRD merged 3 commits into
mainfrom
fix/zig-fd-limit
Sep 4, 2026
Merged

revert(image): go back to glibc until musl can link inside the builder#331
BryanFRD merged 3 commits into
mainfrom
fix/zig-fd-limit

Conversation

@BryanFRD

@BryanFRD BryanFRD commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

The image build has failed on every run since #329 merged, so v1.16.2 went out without one and the security scan reports image not found (package lfsx:1.16.2). This puts a publishable image back and takes the size win off the table until it can be earned in the builder that actually runs it.

What the failure was

zig runs out of file descriptors while linking, on both architectures:

error: sub-compilation of compiler_rt failed
  /opt/zig/lib/std/std.zig:1:1: note: unable to load source: ProcessFdQuotaExceeded

I tried to raise the limit, twice, and the second attempt printed what the first was hiding:

descriptors: 1024 soft, 1024 hard

The hard limit is 1024, so ulimit has nothing to raise it to. The limit belongs to the container the build engine starts, which is the org's reusable workflow rather than this repository, and zig wants more descriptors than that for its own sub-compilations of compiler_rt, libunwind and libubsan.

Why local builds never caught it

Docker Desktop gives a container 1048576 descriptors. I built this Dockerfile here perhaps a dozen times, on both architectures, smoke-tested /health and /ready, checked DNS and TLS against api.github.com, and measured 4.4 MB. All of it was true and none of it could see the one limit that mattered. The lesson is not "test locally first", which I did: it is that a builder difference this specific only shows on the builder.

What this restores

The Dockerfile goes back to the gnu targets and distroless/cc, which is 12.8 MB here rather than 4.4. Built and smoke-tested at --ulimit nofile=1024:1024, the limit that broke the other one.

Two things from the musl work stay, because they were right independently: set -eu on the build step, so a failed build fails its layer instead of exporting an empty one, and everything in #329 outside the Dockerfile (the benchmark measuring the server rather than cargo, the musl bench leg, the arm64 leg of the PR image check).

What happens to #327

Reopened, with this recorded. The next attempt has two candidates and neither is a Dockerfile change: raise nofile where the build container is created, in FerrLabs/.github, or drop zig for a musl cross toolchain that does not spawn a compiler per runtime library. Whichever comes first, the acceptance condition is now a green build in this CI rather than a green build on a laptop.

v1.16.2 stays without an image. The workflow builds from the release tag, so re-running it would rebuild the Dockerfile that fails. The release after this merges carries a working one.

@BryanFRD
BryanFRD enabled auto-merge (squash) September 4, 2026 14:30
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

SonarQube — aucune nouvelle issue

Comparaison entre le projet bac à sable de cette PR et la branche par défaut : SonarQube Community n'analyse pas les PR, ce delta est calculé côté CI. Détail

@ferrfleet ferrfleet Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Diagnosis and the SHELL ["/bin/bash", "-c"] scoping both look right: rust:1.88-bookworm has bash, the directive only affects the one RUN that follows it in this stage, and it doesn't cross the FROM into the distroless stage.

One blocking comment on the ulimit line: the error suppression makes the fix's own failure invisible, and the verification described doesn't cover the build-time condition it targets.

Holding v1.16.2 without an image rather than rebuilding the broken Dockerfile from the tag is the right call.

Comment thread Dockerfile Outdated

@ferrfleet ferrfleet Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The blocking finding from my last review is resolved in 7295b66: the limit is raised to the hard limit rather than a guessed 65536, and the effective value is echoed, so the next CI run will state in the log what zig linked under. That echo is also the verification the description was missing.

One formatting nit inline, not worth holding the merge for.

Comment thread Dockerfile
RUN --mount=type=secret,id=gha-cache-url \
--mount=type=secret,id=gha-runtime-token \
set -eu ; \
ulimit -n "$(ulimit -Hn)" || true ; echo "descriptors: $(ulimit -n) soft, $(ulimit -Hn) hard" ; \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: applying the suggestion swallowed the line continuation, so the ulimit and the echo now share one physical line with a run of spaces between them. It parses fine, but it's the only line in the file that does this:

Suggested change
ulimit -n "$(ulimit -Hn)" || true ; echo "descriptors: $(ulimit -n) soft, $(ulimit -Hn) hard" ; \
ulimit -n "$(ulimit -Hn)" || true ; \
echo "descriptors: $(ulimit -n) soft, $(ulimit -Hn) hard" ; \

@BryanFRD
BryanFRD merged commit fbbdd48 into main Sep 4, 2026
35 of 36 checks passed
@BryanFRD
BryanFRD deleted the fix/zig-fd-limit branch September 4, 2026 14:52
@BryanFRD BryanFRD changed the title fix(image): raise the descriptor limit zig needs to link revert(image): go back to glibc until musl can link inside the builder Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant