platform: make Homebrew packages build, compose, and run on Kandelo#967
Merged
brandonpayton merged 31 commits intoJul 17, 2026
Merged
Conversation
Package promotion currently fails an entire prepare-merge run when a single GitHub release read transiently reports that an existing staging release is missing. Retry exact asset downloads in private attempt directories and publish only a complete archive into the promotion directory. Add focused fixture coverage for transient and permanent failures, partial files, exact arguments, and unsafe input.
Package names and versions may both contain hyphens, so build-index cannot recover their boundary from the transport filename. The old last-hyphen rule silently changed spidermonkey-node@140.11.0esr-node.1 into a different package identity. Carry the validated archived manifest through index composition and validate the transport label with the same canonical renderer used by archive-stage. Fail closed when recovery sees multiple immutable archives for one package and target architecture, reporting both cache and archive identities instead of overwriting by directory order. Keep the change in package publication tooling: package cache keys, archive bytes, and the Kandelo ABI remain unchanged.
Allow conventional owner/homebrew-name repositories to use the reusable publisher while keeping checkout, GHCR, and token authority bound to the caller repository. Carry the distinct canonical owner/name Homebrew identity through build evidence, OCI receipts, Formula paths, and sidecars, with the existing first-party naming exception explicit.
Formula builds run as a separate system identity that cannot traverse the runner checkout parent. Bind the reviewed Kandelo and tap trees into each transient service through root-owned read-only aliases, hide their original paths, and fail explicitly when source auditing cannot traverse a protected tree.
Homebrew-built application images need an existing platform root filesystem, but the VFS builder previously could only pour bottles into an empty image. Accept an explicit ABI-matched platform base while preserving truthful bottle and filesystem provenance. Restore the base with its recorded growth ceiling, rebase only for an explicitly different capacity, and reject unlabeled, ABI-mismatched, or previously Homebrew-composed inputs. Bind only the base hash, byte count, and ABI into output metadata while retaining full source metadata in the external report, so large attestations cannot overflow output metadata or claim the mutated bytes. Record canonical tap identity, preserve unchanged base inodes during default composition, and cover first-party and third-party taps, near-limit metadata, composition rejection, retained files, and both capacity paths.
The publisher rejects repository and tap-name pairs that do not describe the same Homebrew tap, but the VFS planner previously validated those fields only in isolation. Mixed metadata could therefore present a third-party repository as the protected first-party tap during image composition. Apply the same canonical repository-to-tap mapping before any link manifest is loaded. Cover a valid conventional third-party tap plus protected-name aliases, mismatched names, and repositories without the required homebrew- prefix.
Homebrew bottle publication currently proves individual packages, but it does not prove that a dependency-bearing Brewfile closure composes into one exact VFS image and runs unchanged in both Node and Chromium. That leaves registry-backed or single-bottle evidence easy to mistake for the migration acceptance rung. Add an opt-in, tap-owned acceptance selection. The verifier requires a current public GHCR closure with a dependency edge owned by the selected consumer, separates Kandelo platform inputs from Homebrew package evidence, boots the exact composed bytes in Node, and passes the same bytes to a dedicated Chromium runner. A sealed caller can require the gate; ordinary invocations without tap config continue while explicitly producing no closure evidence. The workflow trust seal covers matrix binding, dry-run exclusion, exact tap path confinement, no-fallback public sources, and exact image digests. Focused fixtures cover positive composition plus unrelated-edge, fallback-source, non-root, and symlink rejection. No ABI or package artifact format changes.
Implement Linux AT_EMPTY_PATH semantics for both open descriptors and AT_FDCWD, return ENOENT when the flag is absent, and reject unsupported fstatat/statx flags without rejecting statx synchronization modes. Add kernel coverage and wasm32/wasm64 guest ABI regression tests.
Kandelo permits undefined Wasm imports, so ordinary link probes can report unavailable libc functions as present. Centralize verified musl availability, gnulib string behavior, and the libdl search exception in config.site while preserving caller overrides. Document and test the shared target contract.
Remove Wget's obsolete package-local group_member stub now that the SDK config.site reports the target function truthfully. Bump the package revision because the source and resulting archive change.
Forward -pthread to Clang so configure probes receive the standard _REENTRANT definition while keeping -lpthread as the existing musl compatibility no-op. Keep the source and in-guest SDK drivers aligned and cover the Autoconf compile-and-link shape.
GNU build systems need an accepted musl host tuple to select the libc behavior provided by Kandelo's sysroot. Keep LLVM code generation on wasm-unknown-unknown and use wasm-unknown-linux-musl only for Autoconf host identity, without defining __linux__ or promising Linux kernel behavior.
GNU Netcat 0.7.1 ships config.sub and config.guess files that predate Wasm musl tuples. Replace both from the pinned Automake build input before configure, let config.guess detect the real POSIX build host, and bump the package revision for the changed output.
WebAssembly has no shadow-stack guard page. wasm-ld's roughly 64 KiB default lets deep ordinary call chains overwrite adjacent .bss state, including Kandelo's pthread and TLS globals, before failing later in unrelated code. Reserve 8 MiB, matching the common Linux main-thread default, while preserving any valid larger stack requested by the caller. For Node-hosted commands, first ask the pinned Clang or Clang++ driver for its real -### job graph without injecting SDK glue. Compiler-only modes such as -fsyntax-only, -M, -MM, and --analyze then bypass executable preparation even when they arrive through response files. Confirmed executable links get a second trace with the complete SDK inputs; exactly one pinned wasm-ld job is required, while the wrapped toolchain's expected wasm-opt post-link job remains valid. The Kandelo-native driver applies the same job-graph classification before its direct-link path. Compute the effective stack floor from the exact pinned wasm-ld argv. Relative lld responses use the source-proven driver working-directory slot in LLVM 21's cc1 command. Response expansion is iterative rather than depth-limited: both drivers reject missing files, active canonical cycles, or explicit file/token/text resource limits before linking, and inspect 100-level LLVM-accepted chains without silently lowering a larger request. The Node path decodes LLVM-compatible UTF-8 and BOM-marked UTF-16; the Kandelo-native driver rejects UTF-16 explicitly because Bash cannot retain embedded NUL bytes and the packaged SDK declares no transcoder. Cover both drivers with parser, malformed-input, response-file, fail-closed trace, and real LLVM tests. Direct and response-file forms of all four compiler-only modes run through the actual Node and Kandelo-native entrypoints, with their outputs confined to temporary directories. Executable coverage includes interleaved optimization, include-path, PCH, framework, working-directory, deep-chain, cycle, and alias-cycle cases. The Node path pays for two non-compiling Clang traces per executable link and one for an otherwise-unrecognized compiler-only mode; the native path pays for one classification trace before a possible link. No ABI bump is required because this changes user-program link layout only; the host reads each module's exported stack and heap positions dynamically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Map SDK-owned glue and sysroot paths to fixed identities in file, debug, and macro metadata. Package build systems remain responsible for mapping their own source and dependency paths. This removes checkout-specific Kandelo paths from linked Wasm debug sections without changing the ABI.
Restore musl's basename derivation in the Wasm __libc_start_main overlay so program_invocation_short_name reflects argv[0] without its directory. Programs such as procps pkill use that standard libc identity to select behavior. Extend the exec regression fixture to cover both the full and short program invocation names.
The program builder runs under Bash 3.2 on macOS, where expanding an empty array with nounset enabled aborts the script. Guard the optional wasm64 compiler flags with the same array-preserving expansion already used by the wasm32 path.
Phase B-1 matrix build status —
|
| Package | Arch | Status | Sha |
|---|---|---|---|
| icu | wasm32 | built | e164f41d |
| libcurl | wasm32 | built | b76150ca |
| libcxx | wasm32 | built | 2a4be686 |
| libcxx | wasm64 | built | fd965bda |
| libiconv | wasm32 | built | 07f44851 |
| libpng | wasm32 | built | 7f40fd25 |
| libxml2 | wasm32 | built | 97c1c138 |
| libzip | wasm32 | built | d7331762 |
| openssl | wasm32 | built | 689aff67 |
| openssl | wasm64 | built | dec71db1 |
| sqlite | wasm32 | built | c8d07754 |
| sqlite | wasm64 | built | 2ee96277 |
| zlib | wasm32 | built | 8145a691 |
| zlib | wasm64 | built | 5abb520f |
| bc | wasm32 | built | 8ae7c62c |
| bzip2 | wasm32 | built | 16c33910 |
| coreutils | wasm32 | built | bdfa8ebc |
| curl | wasm32 | built | 57d66c5a |
| dash | wasm32 | built | 2c7f5f40 |
| diffutils | wasm32 | built | 5db7c1a6 |
| dinit | wasm32 | built | e0a980f9 |
| fbdoom | wasm32 | built | 009f7bc6 |
| file | wasm32 | built | 4f5db169 |
| findutils | wasm32 | built | 4db3eb1a |
| gawk | wasm32 | built | c8e5b975 |
| git | wasm32 | built | 052cc889 |
| grep | wasm32 | built | 88eacae8 |
| gzip | wasm32 | built | 40ff8b0b |
| hello | wasm32 | built | 4021a41c |
| kandelo-sdk | wasm32 | built | c6e2160e |
| kernel | wasm32 | built | 1a06bb50 |
| less | wasm32 | built | 0e2a8fc4 |
| lsof | wasm32 | built | 438699b8 |
| m4 | wasm32 | built | 3238e030 |
| make | wasm32 | built | 495dbe97 |
| mariadb | wasm32 | built | d5cc9820 |
| mariadb | wasm64 | built | 7a8ff073 |
| modeset | wasm32 | built | 82441354 |
| msmtpd | wasm32 | built | 647518a5 |
| nano | wasm32 | built | 2550bfb3 |
| ncurses | wasm32 | built | 6e17c2c9 |
| netcat | wasm32 | built | 9e8ecc3f |
| nginx | wasm32 | built | 5b6b89a1 |
| php | wasm32 | built | 6f290cba |
| posix-utils-lite | wasm32 | built | abe967f0 |
| ruby | wasm32 | built | 73b79fc6 |
| sed | wasm32 | built | 53d99b3a |
| spidermonkey | wasm32 | built | b96a5e5d |
| tar | wasm32 | built | 8b977566 |
| tcl | wasm32 | built | d610c8d8 |
| unzip | wasm32 | built | f4f0f7fb |
| userspace | wasm32 | built | 3ece4153 |
| vim | wasm32 | built | dca7a50e |
| wget | wasm32 | built | a0e34357 |
| xz | wasm32 | built | 7b73a2d1 |
| zip | wasm32 | built | 9c1192b2 |
| zstd | wasm32 | built | d54d78b2 |
| bash | wasm32 | built | cb3ae7ce |
| mariadb-test | wasm32 | built | ccbc5280 |
| mariadb-vfs | wasm32 | built | d3572eee |
| mariadb-vfs | wasm64 | built | dffa9e4b |
| nethack | wasm32 | built | 3ab3eb69 |
| node | wasm32 | built | 6f99f184 |
| spidermonkey-node | wasm32 | built | b8bc91f5 |
| vim-browser-bundle | wasm32 | built | f1ba7ece |
| nethack-browser-bundle | wasm32 | built | e36e7d39 |
| rootfs | wasm32 | built | f490bbd5 |
| shell | wasm32 | built | 63ddc687 |
| lamp | wasm32 | built | e4d5aaea |
| node-vfs | wasm32 | built | a9a23e15 |
| wordpress | wasm32 | built | 9a0ad3c8 |
Auto-generated; replaced on each push. Raw data in the publish-status workflow artifact.
Nonzero exit codes are normal POSIX process disposition, not host-runtime failures. Stop emitting status-derived diagnostics and process-worker console noise on both hosts while retaining typed, context-rich diagnostics for actual browser worker traps and crashes. Add process-boundary Node coverage, real Chromium exit/trap coverage, routing assertions, and architecture documentation. This does not change the Kandelo ABI.
Apply the reviewed Kandelo bottle-tag patch to a pinned Homebrew revision and record reproducible source provenance. Generate a root-owned system environment that makes the guest architecture tag authoritative over prefix and user configuration while keeping the canonical Homebrew launcher path. Test exact tap bottle selection, patch drift and unsafe Git-store rejection, cross-timezone archive identity, hostile environment precedence, and the truthful ABI 39 runtime boundary. This does not change the Kandelo ABI.
The kernel normalizes channel failures to CH_RETURN=-1 and stores the actual error in CH_ERRNO. Forward that errno through every intercepted semctl command shape instead of deriving EPERM from the normalized return value. Extend the SysV IPC guest regression to verify EINVAL after IPC_RMID for IPC_STAT, GETALL, SETALL, and scalar GETVAL.
Track endpoint references owned by queued descriptors separately from externally owned readers. Successful recvmsg transfers each reference to its new OFD, while unusable control buffers and receiver allocation failures release it exactly once. Install the descriptor prefix that fits a short control buffer, close the excess, and report MSG_CTRUNC. Mark ancillary queues reachable from live receive endpoints and follow queued socket rights as graph edges, then sweep unrooted self and mutual cycles. Add kernel slot-reuse coverage and a real Wasm regression for read/write endpoint transfer, live and abandoned socket-rights cycles, absent and truncated control buffers, and partial EMFILE installation. This changes no ABI structure or version.
mkfifo and mknod(S_IFIFO) previously created ordinary host files, so readers observed premature EOF instead of FIFO rendezvous semantics. Implement marker-backed, kernel-owned named pipes through the normal VFS and descriptor paths. Preserve identity and metadata across hard links, rename, unlink, and recreation; reserve descriptor slots across blocking opens; cancel exact reservations on signals, thread exit, exec, and process removal; keep O_PATH/O_SEARCH handles path-only; and exercise fork, exec, threads, cancellation, Node, and Chromium behavior. Serialize exact FIFO path-only, read-only cohort, write-only, and read-write ownership through SCM_RIGHTS. Sender close, receive, discard, control truncation, EMFILE rollback, fork, close, and teardown now transfer or release the same reference kind exactly once. The structural ABI snapshot is unchanged. The musl SA_RESTART correction still requires rebuilt guest binaries in the next ABI package wave.
brandonpayton
force-pushed
the
integration/homebrew-abi41-final-after934-20260716
branch
from
July 17, 2026 01:14
f7161ba to
7f6d14a
Compare
brandonpayton
added
the
ready-to-ship
|
prepare-merge: test-gate passed against the synthetic PR merge and sealed |
brandonpayton
deleted the
integration/homebrew-abi41-final-after934-20260716
branch
July 17, 2026 02:30
This was referenced Jul 17, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Kandelo users cannot yet rely on one complete path from a Homebrew Formula to a published bottle (a prebuilt package archive), a Brewfile-built virtual filesystem (VFS) image, and a program running inside Kandelo. The missing work spans publication trust, reproducible image composition, cross-compilation facts, guest Homebrew startup, and POSIX behavior exposed by real packages.
Merging the application binary interface (ABI) changes separately would rebuild Kandelo's enabled ABI-bound package set more than once. This PR keeps each fix as its own commit for review and debugging, while giving CI one final ABI 41 tree to build and test.
This matters to package authors and users who need the default tap, future third-party taps, Node.js hosts, and browser hosts to observe the same package and runtime behavior.
What changed
fstatat, named pipes, passed-pipe lifetime, advisory locks, procfs descriptor metadata,dlopenmain scope, locale reporting, guest output writes, and ordinary exit diagnostics./usr/bin/brewas a direct symlink while teaching the reviewed Homebrew patch to retain the canonical/home/linuxbrew/.linuxbrewprefix for that exact alias.The 31 commits remain separate and in their original order. Compared with the previously staged branch, commits 1 through 29 are byte-for-byte unchanged, commit 30 is the corrected ABI 41 reserve, and commit 31 is the separate non-ABI Homebrew alias fix.
ABI and artifacts
The final source tree is ABI 41. The shared snapshot, C header, and generated TypeScript bindings all declare ABI 41.
The already-built 71 ABI 41 candidate archives remain valid. Instrumented programs receive their fork-buffer address from the host at runtime; neither the address nor the reserve capacity is baked into those programs. The reserve change also does not alter package cache-key inputs. CI may therefore verify and reuse those archives instead of rebuilding them.
The alias fix changes the patched Homebrew source archive, its provenance, and the bootstrap VFS image. It does not change Kandelo package archives.
Existing ABI 39 artifacts remain intentionally incompatible. This PR adds no stale-artifact compatibility path. A continuation can still exceed the fixed 60 KiB reserve; the existing host detector terminates that process truthfully, and pre-write bounds enforcement remains tracked in #922.
Eight registry packages remain explicitly disabled in both staging workflows because their builds were previously removed from CI: CPython, Erlang, Erlang VFS, Perl, Perl VFS, Python VFS, Redis, and Texlive. None has an entry in the current ABI 39 binary index, so this ABI bump does not turn a working published archive into an outage. They must receive supported ABI 41 builds or complete their Homebrew Formula migration before being described as available.
Validation
Base:
62089b18ca237cd8df4b1f18f765c8187957f03cCurrent head:
7f6d14a45d8bc5b137601fb0563c8dfd76ea0710Current tree:
b133bfa018f7e2bc84c4845edf2f945b2f844954The previous exact-head staging run reused all 71 ABI 41 archives and passed kernel, fork-instrument, POSIX, both libc shards, all three Sortix shards, full Vitest, and Chromium.
Its Prepare Merge run content-verified and promoted those archives, then found a real guest blocker: Homebrew's Bash continuation needed 49,232 bytes while the 48 KiB draft reserved 49,152. That run is evidence for the diagnosis, not merge authority for the new head.
Run locally on the current exact head through
scripts/dev-shell.sh:scripts/check-abi-version.shpassed; the snapshot, C header, TypeScript bindings, and ABI bump agree.dlopenVitest passed: 4 files, 48 tests passed, 17 skipped.--prefix,--repository, and--version.brew --versionpassed inside Kandelo through both/home/linuxbrew/.linuxbrew/bin/brewand the direct/usr/bin/brewsymlink, with canonical prefix and repository identities.git diff --checkpassed.The exact-head staging run passed. It verified and reused all 71 ABI 41 archives, scheduled no package builds, rematerialized the candidate workspace, and passed kernel, fork-instrument, Vitest, Chromium, POSIX, both libc groups, and all three Sortix groups.
After rechecking the unchanged head and base,
ready-to-shipwas restored. Prepare Merge completed successfully on synthetic mergef57c53a62bbe33352b2568f94010218b9d116503. It promoted all 71 content-verified archives without package rebuilds, rebuilt the kernel/program/rootfs workspace, passed real Homebrew startup through both the canonical path and/usr/bin/brew, and passed kernel, fork-instrument, Vitest, Chromium, POSIX, both libc groups, and all three Sortix groups. GitHub postedmerge-gate=successfor the current head and sealed the ABI 41 candidate.Merge method
Use rebase merge. The exact-head Prepare Merge gate has succeeded. This preserves the individual commits and allows absorbed source PRs to be closed without losing their changes.