Skip to content

Conversation

@bolinfest
Copy link
Collaborator

@bolinfest bolinfest commented Feb 12, 2026

Problem

rust-release was failing for two independent reasons:

  1. aarch64-unknown-linux-musl failed at link time with /usr/bin/ld: cannot find -lcap when building binaries that transitively pull in codex-linux-sandbox.
  2. The release job failed in Create GitHub Release with Not Found from the release-assets API because files: dist/** included many files named cargo-timing.html, which triggered conflicting delete/upload operations for the same release asset name.

Why this is the right fix

codex-linux-sandbox builds vendored bubblewrap and links libcap. In musl jobs, distro libcap-dev provides host/glibc artifacts, which are not valid for musl cross-linking. Building/staging target-compatible libcap in the musl tool bootstrap and pointing pkg-config at it fixes the linker mismatch directly.

For release assets, softprops/action-gh-release uploads by basename. Allowing multiple cargo-timing.html files into dist/** creates asset-name collisions and racey API behavior. Removing those files from dist/ before release upload prevents the collision.

What changed

  • Updated .github/scripts/install-musl-build-tools.sh to install tooling needed to fetch/build libcap sources (curl, xz-utils, certs).
  • Added deterministic libcap bootstrap in the musl tool root:
    • download libcap-2.75 from kernel.org
    • verify SHA256
    • build with the target musl compiler (*-linux-musl-gcc)
    • stage libcap.a and headers under the target tool root
    • generate a target-scoped libcap.pc
  • Exported target PKG_CONFIG_PATH so builds resolve the staged musl libcap instead of host pkg-config/lib paths.
  • Updated .github/workflows/rust-ci.yml to add a release matrix entry for aarch64-unknown-linux-musl on the ARM runner.
  • Updated .github/workflows/rust-ci.yml to set CARGO_PROFILE_RELEASE_LTO=thin for release matrix entries (and keep fat for non-release entries), matching the release-build tradeoff already used in rust-release.yml while reducing CI runtime.
  • Updated .github/workflows/rust-release.yml release cleanup to remove cargo-timing.html files from dist/ before Create GitHub Release (files: dist/**).

Verification

  • Reproduced the original musl failure in CI-like containers:
    • aarch64-unknown-linux-musl failed with cannot find -lcap.
  • Verified the underlying mismatch by forcing host libcap into the link:
    • link then failed with glibc-specific unresolved symbols (__isoc23_*, __*_chk), confirming host libcap was unsuitable.
  • Verified the musl fix in CI-like containers after this change:
    • cargo build -p codex-linux-sandbox --target aarch64-unknown-linux-musl --release -> pass
    • cargo build -p codex-linux-sandbox --target x86_64-unknown-linux-musl --release -> pass
  • Triggered rust-ci on this branch and confirmed the new job appears:
    • Lint/Build — ubuntu-24.04-arm - aarch64-unknown-linux-musl (release)
  • Confirmed the release-asset failure mode in GitHub Actions for rust-release run 21938547951 (release job), where duplicate cargo-timing.html assets caused Not Found responses from release asset update/delete API calls.

Copy link
Contributor

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a9416d588

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


# Allow pkg-config resolution during cross-compilation.
echo "PKG_CONFIG_ALLOW_CROSS=1" >> "$GITHUB_ENV"
echo "PKG_CONFIG_PATH=${libcap_pkgconfig_dir}\${PKG_CONFIG_PATH:+:\${PKG_CONFIG_PATH}}" >> "$GITHUB_ENV"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Expand PKG_CONFIG_PATH before writing to GITHUB_ENV

This writes a literal \${PKG_CONFIG_PATH:+:\${PKG_CONFIG_PATH}} into $GITHUB_ENV instead of expanding it, so subsequent GitHub Actions steps receive a malformed PKG_CONFIG_PATH containing ${...} text rather than a real colon-joined path. Because GITHUB_ENV values are imported as-is, plain pkg-config consumers in later steps can miss the staged musl libcap.pc (and any prior path entries), which undermines the cross-linking setup this change is trying to enforce.

Useful? React with 👍 / 👎.

Problem:
The `aarch64-unknown-linux-musl` release build was failing at link time with
`/usr/bin/ld: cannot find -lcap` while building binaries that transitively pull
in `codex-linux-sandbox`. CI did not include an arm64 linux musl representative
release build in `rust-ci`, so this release-only target regression was not
caught before release.

Why this is the right fix:
`codex-linux-sandbox` compiles vendored bubblewrap and links `libcap`. In the
musl jobs, we were installing distro `libcap-dev`, which provides host/glibc
artifacts. That is not a valid source of target-compatible static libcap for
musl cross-linking. The correct fix is to produce a target-compatible libcap
inside the musl tool bootstrap and point pkg-config at it.

Separately, adding arm64 linux musl to the representative release matrix in
`rust-ci` ensures this class of release-only breakage is caught by normal CI.
Using `thin` LTO for those representative release-profile checks keeps CI
feedback fast while still exercising release-profile behavior.

What changed:
- Updated `.github/scripts/install-musl-build-tools.sh` to install tooling
  needed to fetch/build libcap sources (`curl`, `xz-utils`, certs).
- Added deterministic libcap bootstrap in the musl tool root:
  - download `libcap-2.75` from kernel.org
  - verify SHA256
  - build with the target musl compiler (`*-linux-musl-gcc`)
  - stage `libcap.a` and headers under the target tool root
  - generate a target-scoped `libcap.pc`
- Exported target `PKG_CONFIG_PATH` so builds resolve the staged musl libcap
  instead of host pkg-config/lib paths.
- Added `ubuntu-24.04-arm` + `aarch64-unknown-linux-musl` + `profile: release`
  to the representative release builds in `.github/workflows/rust-ci.yml`.
- Configured `rust-ci` to use `CARGO_PROFILE_RELEASE_LTO=thin` for matrix
  entries where `profile == release`.

Verification:
- Reproduced the original failure in CI-like containers:
  - `aarch64-unknown-linux-musl` failed with `cannot find -lcap`.
- Verified the underlying mismatch by forcing host libcap into the link:
  - link then failed with glibc-specific unresolved symbols
    (`__isoc23_*`, `__*_chk`), confirming host libcap was unsuitable.
- Verified the fix in CI-like containers after this change:
  - `cargo build -p codex-linux-sandbox --target aarch64-unknown-linux-musl --release` -> pass
  - `cargo build -p codex-linux-sandbox --target x86_64-unknown-linux-musl --release` -> pass
@bolinfest bolinfest enabled auto-merge (squash) February 12, 2026 07:57
@bolinfest bolinfest merged commit 08a0008 into main Feb 12, 2026
33 checks passed
@bolinfest bolinfest deleted the pr11556 branch February 12, 2026 08:08
@github-actions github-actions bot locked and limited conversation to collaborators Feb 12, 2026
@bolinfest bolinfest changed the title Fix linux-musl release link failures caused by glibc-only libcap artifacts Fix rust-release failures in musl linking and release asset upload Feb 12, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants