Skip to content

ci: resolve full Rust toolchain from manifest - #430

Merged
andrei-hasna merged 6 commits into
mainfrom
fix/rust-ci-full-toolchain-disk
Jul 28, 2026
Merged

ci: resolve full Rust toolchain from manifest#430
andrei-hasna merged 6 commits into
mainfrom
fix/rust-ci-full-toolchain-disk

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • provision cross-target Rust components from codex-rs/rust-toolchain.toml instead of the pinned action default
  • apply the same manifest-driven target setup to nextest archive and shard jobs
  • disable workspace-local sccache only for the Linux remote nextest archive that exhausted hosted-runner disk

Run diagnosis

  • jobs 90145854922, 90145854944, 90145854941, 90145855047 and release variants 90145854991, 90145855003, 90145855014 installed targets for Rust 1.95.0 while Cargo selected 1.97.1, producing E0463 missing core
  • job 90145855045 selected the local sccache fallback, then the runner crashed during archive compilation with No space left on device
  • other clippy, argument-comment-lint, and nextest test failures are source/test failures outside this workflow-only change

Local validation

  • workflow YAML parse and focused matrix/cache assertions
  • bunx prettier@3.5.3 --check on both changed workflows
  • .github/scripts/test_rust_ci_full_nextest_platform_workflow.py
  • Rust 1.97.1 manifest resolution and target installation from codex-rs
  • actionlint YAML/expression validation (full ShellCheck reports only pre-existing findings outside changed blocks)
  • staged and exact-base branch-range gitleaks scans

Exact base: 2968352abd33e7bf6e6a3fd1024b52fd481299da
Todos: bd39cff0-226e-47a4-8275-82ad121bff22


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Updated exact-head evidence after the first disk mitigation failed:

  • 8f9060a88479582fae83a37f352c6faf464229af is not mergeable: full run 30322801070, job 90161987805, completed the 22-minute build but failed archive creation with 7 MB free and ENOSPC.
  • New head: 1c94d79b58f2dd9fafdcbd9adefc9f1c1137868f.
  • Revised archive fix: only the Linux remote archive producer uses blacksmith-16vcpu-ubuntu-2404; shards remain on ubuntu-24.04, sccache is retained, and full matrix coverage is unchanged.
  • Added exact workflow regression in .github/scripts/test_rust_ci_full_nextest_platform_workflow.py.
  • Local YAML, Prettier, actionlint expression checks, Python tests, diff checks, and staged/range gitleaks passed.
  • Exact-head Testbox 30324298525 passed, observing a 738 GB filesystem with 640 GB free plus Rust 1.97.1 manifest resolution and target installation.
  • New full matrix 30324367725 was dispatched only after Testbox and is pending.

No merge requested.

@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Final-head update after Blacksmith archive compatibility fix:

  • Current head: d196233c7e48b168ea2c63ab6645b6760c349c23.
  • 30324367725 was cancelled as superseded after the capacity runner exposed an archive-only setup bug: the archive producer attempted the shard-only unprivileged-userns sysctls.
  • The archive job is now build-only; Linux shard jobs retain user-namespace setup before executing tests. An exact regression asserts that separation.
  • Final-head Testbox 30324525338 passed three workflow regressions, Rust 1.97.1 manifest resolution, target installation, and the >=500 GiB capacity gate (738 GB filesystem, 640 GB free).
  • Final full matrix 30324592235 is active. Its Linux archive job 90167194670 is on blacksmith-16vcpu-ubuntu-2404, passed manifest toolchain setup, skipped the removed sysctl step, and is building the nextest archive.

No merge requested.

@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Exact-head update: 717d3b2bb0f60e801fea608f01b8904f237f3ede supersedes d196233c7e48b168ea2c63ab6645b6760c349c23, which remains not mergeable because run 30324592235 exhausted GitHub-hosted shard disks while extracting the 14.56 GB Linux remote archive.

This revision keeps the proven manifest-toolchain and Blacksmith archive changes, routes both the Linux remote archive and all four shards through the caller's single runner: blacksmith-16vcpu-ubuntu-2404 input, preserves remote_env: true, and guards shard-only namespace sysctls by their /proc/sys nodes. The exact regression failed on the old head and now passes locally. Staged and exact-base branch-range gitleaks were clean before commit and push.

Next gate is exact-head Blacksmith Testbox. A new full rust-ci-full run will be dispatched only after Testbox passes.

@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Workflow/disk proof at exact head 717d3b2bb0f60e801fea608f01b8904f237f3ede:

  • Blacksmith Testbox 30341369121 passed (job 90217496211): exact checkout, three workflow regressions, 640 GiB free, guarded user namespaces with unshare/bwrap, and manifest-selected Rust target provisioning.
  • Full rust-ci-full run 30341508344 reached the complete Linux remote path. Archive job 90217942506 succeeded on Blacksmith and uploaded a 14,559,948,983-byte nextest artifact.
  • Shard 1 90221636657, shard 2 90221636588, shard 3 90221636603, and shard 4 90221636550 all ran on 16-vCPU Blacksmith runners. For every shard, guarded namespace setup, unchanged remote Docker env setup, and nextest archive download/extraction succeeded; each then entered and completed its actual tests step.
  • No shard log contains No space left, error extracting archive, or failed to unpack. The ENOSPC failure is resolved.

The shard job conclusions remain red only for product/environment tests outside this workflow-only PR (shell snapshot/sandbox expectations, app-server worker-process cleanup, external-agent-session import, MCP approval, and related permission tests). No workflow-specific failure was found, and no product code was changed.

Per owner direction, PR #430 stays open and clean at this head. The required all-green shard proof is deferred until this branch is rebased/rerun on combined main after PR #433 and the app-server remediation land; Peirce is not resumed before that stable combined candidate. No merge.

@andrei-hasna
andrei-hasna force-pushed the fix/rust-ci-full-toolchain-disk branch from 717d3b2 to e0277a1 Compare July 28, 2026 14:35
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Rebased exact-head candidate: e0277a1cc8a04f06651a4d5826018e4c5db975a3 on exact main a8a2d4a9a49073982174e11d76dfb372aee559aa after PRs #431-#435.

  • All four existing commits replayed without conflicts.
  • git range-diff marks each old/new commit patch-identical.
  • Diff remains limited to .github/workflows/rust-ci-full.yml, .github/workflows/rust-ci-full-nextest-platform.yml, and .github/scripts/test_rust_ci_full_nextest_platform_workflow.py.
  • Workflow regressions, YAML parse, Python compilation, Prettier, actionlint expression validation, and diff checks pass. Full actionlint has only the unchanged pre-existing SC2086/SC2317 findings outside the owned edits.
  • Staged and origin/main..HEAD gitleaks scans are clean; force-push used an explicit lease on old head 717d3b2b.

Next gate is exact-head Blacksmith Testbox. No full rust-ci-full dispatch or reviewer recheck occurs before Testbox passes.

@andrei-hasna
andrei-hasna merged commit cebbce3 into main Jul 28, 2026
50 of 65 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant