Roadmap sweep: v1.x candidates, v2 directions, packaging and CI gaps - #3
Draft
allandng wants to merge 17 commits into
Draft
Roadmap sweep: v1.x candidates, v2 directions, packaging and CI gaps#3allandng wants to merge 17 commits into
allandng wants to merge 17 commits into
Conversation
…stays provisional 50 MB blob on a 4-vCPU cloud container: 62.3us median (p99 97.1us), 101x/355x over UDS/HTTP, 5.5 GB/s on the 16 KB stream. Labeled virtualized — bare metal remains the missing piece for the headline claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
.clang-format is derived from the existing style (near-no-op on the tree; byte-identical on the core headers). CONTRIBUTING documents the frozen-ABI, frozen-layout, unknown-flag-bits, and platform-seam rules plus the two-platform harness. CHANGELOG seeds 1.0.0/1.1.0 retrospectively with an honest note about the missing tags. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
…-config project(VERSION 1.1.0); SHUTTLE_BUILD_TESTS/SHUTTLE_INSTALL options defaulting to top-level-only so add_subdirectory/FetchContent consumers get just the two libraries; shuttle::core / shuttle::c namespaced targets exported with a config package (SameMajorVersion); libshuttle_c.so.1 SONAME frozen to ABI v1; shuttle.pc for non-CMake consumers. Sanitizer target ordering preserved and verified (zero __asan symbols in shuttle_c/bench/perf binaries). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
Native Apple clang runs of the same correctness suite with the same perf-measurement exclusions as the Linux jobs. Notes in the yml cover the os_sync_wait_on_address 14.4+ image requirement and flake triage order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
…C ABI bindings/python: shuttle-ipc (cffi) — Channel with blocking/nonblock/polling- timeout writes, zero-copy BorrowedMessage guard that invalidates on release, ShuttleError hierarchy; 28 pytest cases green against a fresh libshuttle_c.so. bindings/rust: shuttle-sys (no_std hand-written externs, no bindgen) + shuttle safe wrapper — Producer/Consumer split, Borrowed<'a> making use-after-release E0597 and double-acquire E0499 compile errors (verified); 20 integration tests + 3 doc-tests green. Existing tests/ffi conformance tests untouched. Neither package is publish-configured; both consume only shuttle_c.h v1.1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
Every syscall that creates, sizes, maps, unmaps, or destroys the shared segment now goes through seg_create/seg_open/seg_size/seg_map/seg_unmap/ seg_close/seg_unlink, with a SegBacking enum (kShm only today). New backings — explicit hugetlbfs, a Windows file mapping — arrive as enum values plus a branch in the seam, never a syscall in src/. No behavior change: error mapping is expression-identical, one-shot sizing preserved, ASan and TSan suites 24/24. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
SHUTTLE_CREATE_STATS creates a kVersionStats=2 segment: two cache lines of single-writer counters (msgs/bytes written, read; dropped reserved) appended after the park block. Every v1 offset is pinned by static_asserts, and kDataOffsetV1 is derived independently from the park-block end so the append-only claim is proven, not assumed. Plain creates keep writing v1 segments — the default on-disk format is unchanged; old binaries opening a v2 segment get BAD_VERSION from their exact version check. Counters are relaxed single-writer load+store (heartbeat idiom), gated by a pointer resolved once at Producer/Consumer construction — null on v1, where those bytes are the data region. shuttle_get_stats + SHUTTLE_ERR_NO_STATS land additively; SHUTTLE_ERR_NO_HUGEPAGES and the hugetlb flag bits are pinned as reserved for the next package. inspect prints the stats block on v2 segments only. New stats_test (6 cases incl. version/geometry rejection with demonstrated falsifiability and a TSan-visible concurrent watcher); suites 25/25 ASan and TSan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
…1GB) Guaranteed huge-page backing as a segment file on a hugetlbfs mount, discovered via /proc/mounts by pagesize. Any failure to deliver — no mount, no free pages (ENOMEM at mmap, where hugetlbfs actually reserves), macOS — returns SHUTTLE_ERR_NO_HUGEPAGES (-14), never a silent fallback to normal pages; that is the contract difference from the advisory THP flag. Openers need no special mapping flags: the hugetlbfs inode dictates the page size (verified cross-process via smaps KernelPageSize). Sizing rounds up to the huge page; data_capacity keeps the caller's exact value. Dual-namespace rule documented: shm wins on open. Both error and positive paths were exercised: this box permitted a temporary vm.nr_hugepages=64 + hugetlbfs mount, so cross-process roundtrip, stats+ hugetlb and THP+hugetlb combos, unlink cleanup, and the mount-present/ zero-pages -14 path all ran, then the box was restored. Suites 26/26 ASan and TSan in the restored (no-hugepage, SKIP-path) state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
A strictly per-call lossy write mode: implies try-semantics (never parks, even against a parked or dead consumer); on a full ring the message is discarded, counted in stat_msgs_dropped on stats segments, and the call returns SHUTTLE_DROPPED (1) — the ABI's first positive return, so the docs now recommend testing errors as rc < 0. No cursor moves, nothing queued is touched: a drop is invisible to the peer. The flag is rejected with INVALID_ARGS on read/acquire paths. Default behavior is bit-for-bit unchanged — never-drops remains the v1 guarantee. Overwrite-oldest is recorded in the roadmap as rejected for v1.x: it would break single-writer cursor ownership, could invalidate a live borrow, and races the consumer's frame parsing. Consumer drain-to-latest is documented as the sound freshness pattern instead. New drop_policy_test (7 cases; falsifiability executed: an injected silent-loss bug is caught by five assertions). Suites 27/27 ASan and TSan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
header_fuzz found a real bug in ~2s: open()'s geometry check used the uint64 sum `max_payload + kFrameHeader > data_capacity`, which wraps for a max_payload near 2^64 — so open() accepted a segment claiming an 18-exabyte max_payload, disarming Consumer::parse's length guard and handing a forged out-of-bounds span to the caller. Fixed by extracting the magic/version/ geometry checks into a pure validate_header() (reachable by the fuzzer) with every bound written as subtraction against a checked floor. Closed the same overflow shape in create()'s FR-4 check and parse()'s backstop so the class is gone, not just its one reachable instance. No legitimate geometry changes verdict; all 27 tests green ASan+TSan, and the minimized crash no longer reproduces (9.7M fuzz execs clean post-fix). Harnesses: bipbuffer_fuzz (op tape vs deque reference model, guard canaries, wrap invariant) and header_fuzz (arbitrary/seeded/field-directed segments with overflow-safe soundness oracle). SHUTTLE_FUZZ=ON is clang-only (hard-errors under gcc). New fuzz-smoke CI job runs each 60s from a cold corpus and uploads reproducers on failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
…+ compile-only glue) A separate, additive, opt-in module — does not touch shuttle_c.h, the segment layout, or SHUTTLE_ABI_VERSION. A producer holding GPU-resident data packs a fixed 160-byte little-endian descriptor (opaque cudaIpcMemHandle_t, device, offset/len, optional event handle) and sends it as an ordinary Shuttle message; the consumer unpacks and opens it via CUDA IPC. The host codec (pack/unpack/validate) has no CUDA dependency and is fully tested — byte-exact wire layout, rejection of every malformed image, 200k-iter fuzz-lite with ASan enforcing no over-read. The cudaIpc* glue is behind SHUTTLE_WITH_CUDA, compiled only (against a stub in CI), never run. Honestly scoped in docs/CUDA_DESIGN.md: proven = host codec + well-formed glue; unproven without hardware = cross-process device visibility, event sync, handle-cache refcount, perf, the borrow-vs-kernel lifetime race. New cuda-glue-compile CI job builds+runs the codec test and compile-checks the glue against a stub cuda_runtime.h. Suite +1 (cuda_desc_test). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
Contract for four phases: P1 page-aligned spans (0x10), P2 file-backed channels (path-typed symbols, SegBacking::kFile), P3 WILLNEED prefetch + read-only peek_next, P4 EXPERIMENTS.md and doc sweep. Evidence-ledger answer to the gating question: borrow semantics are strictly release-before-acquire (spsc.hpp try_read/release, shuttle_c.cpp ensure_borrow), so P3's pipelining is a new read-only API, not documentation. Verified with verify_plan.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
…tal) Third platform branch in the seam: CreateFileMappingW/OpenFileMappingW/ MapViewOfFile for segments, WaitOnAddress/WakeByAddressAll for park/wake, QueryPerformanceCounter for time — every _WIN32 stays in platform.hpp. The header's pthread park block becomes a seam-defined ParkArea: on POSIX it is the identical three members at identical offsets — proven by a hard-coded compile-time tripwire (kDataOffsetV1 == 1280, same value on glibc and macOS) plus unchanged offset asserts — on Windows an inert placeholder (waits ride the cursor words, as on macOS). Windows named sections vanish with their last handle, so the Channel retains the section HANDLE (POSIX drops the fd exactly as before) and seg_unlink is existence-report-only there; documented as a parity gap. Heartbeat liveness is the Windows crash story; robust-mutex recovery and the posix_spawn gate suite remain POSIX-only. New windows-latest CI job compiles the libraries + pure-logic tests and runs a threads+CreateProcess smoke; the Windows path is CI-validated only (no Windows toolchain here). POSIX suites: 28/28 ASan and TSan (filtered), byte-identical behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
Opt-in create flag 0x10: every borrowed payload pointer is page-aligned so consumers can wrap spans in MTLBuffer newBufferWithBytesNoCopy or cudaHostRegister without a copy. Aligned framing = header page + payload rounded to page, stated in one place (frame_span/frame_fits helpers beside kFrameHeader, overflow-safe floor form everywhere). data_offset rounds to page under the flag, so an old binary REJECTS an aligned segment at the geometry check (kErrCorrupt) instead of misparsing — deliberate, tested in both directions against a reimplementation of the pre-v1.4 rule. Worst-case internal fragmentation is 2*page-9 bytes/message (8183 at 4 KiB pages; 0.016% of a 50 MB payload), asserted by measurement. Mode is resolved once at Producer/Consumer construction from the flags word; stats counters keep counting payload bytes, not stride (mutation-tested). Also brings the Python and Rust binding packages up to the full ABI surface (stats, hugetlb, drop-newest, aligned): pytest 28->37, cargo 20->29+4. Suites 29/29 ASan and TSan; header_fuzz 10.5M and bipbuffer_fuzz 181k execs clean post-change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
Segments as disk-file mmaps via SegBacking::kFile, so capacity can exceed physical RAM with the page cache handling residency. Path-typed symbols rather than overloading shm names: '/tmp/cache' is a legal shm name too, so which namespace a string belongs to must be a compile-time fact, not a runtime guess. Existing name-based seam paths are byte-for-byte untouched; create/open share one publication/validation implementation with the file variants so the two cannot drift. Crash story re-argued and OBSERVED, not assumed: EOWNERDEAD is returned on a file-backed MAP_SHARED mapping after SIGKILL holding the park mutex, and pthread_mutex_consistent restores full service — proven with a raw-mutex probe plus a negative control (skipping recovery leaves ENOTRECOVERABLE). Heartbeat kill-point behavior identical to shm. Stale-file recovery recipe and the msync-never-called durability non-goal documented. 512 MB streamed byte-exact through a 256 MB channel. Flag 0x20 persisted informational; hugetlb bits rejected on the file path as contradictory. Suites 30/30 ASan and TSan; pytest 48, cargo 36+4 doc-tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
Prefetch: advise_willneed in the seam (POSIX_MADV_WILLNEED / MADV_WILLNEED / Windows no-op), consumer-only, on acquire and just before parking, advising only committed-but-unread data (both runs when wrapped). Gated by a bool resolved once at Consumer construction — true only for file-backed segments — so the default path pays one predictable branch; addresses floored to page boundaries (never escaping the mapping, whose base is page-aligned). peek_next: reports whether the next UN-borrowed message (position = read + borrowed_) is committed and its payload length, using try_read's existing P1/P2 acquire edges — no new writer, no new ordering edge, documented inline in the contract block (peek can under-report via staleness, never over-report). The wrap case reads the header at offset 0 WITHOUT the C2 handoff store, so read keeps a single writing site; the falsifiability demo (a peek that performs the handoff) is caught by the cursor-unmoved assertion. C ABI: shuttle_peek_next (OK/WOULD_BLOCK/INVALID_ARGS/CORRUPT). Rust keeps both compile-time guarantees by pairing Consumer::peek_next(&self) with Borrowed::peek_next, pinned by a compile_fail(E0502) doctest. Suites 31/31 ASan and TSan; excluded wakeup/perf tests explicitly re-run green (trickle 100k msgs, 0 slow reads); pytest 52, cargo 39+6 doctests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
…herence docs/EXPERIMENTS.md: dated, numbered measurements on the virtualized host — E1 bench rerun (63.5us median, no default-path drift from v1.4; p99 flagged as noisy), E2 aligned-vs-classic (fragmentation exactly as predicted, 4KiB speedup measured but unexplained and explicitly not sold), E3 file-vs-shm latency parity at stable sizes, E4 prefetch cold-cache NULL RESULT recorded honestly (host cache defeats guest eviction; hint documented as not a speedup), E5 acquire_read idempotence verified. Closes with the KV-cache orchestrator/sidecar worked example citing the entries above. README reflects the whole branch: Since-v1.1 capability list, drop-newest amendment to the never-drops bullet, install/find_package/pkg-config and bindings consumption paths, 36-test count, fuzz harnesses, updated layout, E1 re-measurement in Benchmark honesty. API.md: v1.4 version block and the acquire_read doc corrected to the implemented idempotent behavior (verified against the built library; noted as a doc change). CHANGELOG [Unreleased] backfills all 16 branch commits with the validate_header overflow called out as security-relevant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two passes, landed as one reviewed commit per work package: the full docs/ROADMAP.md triage plus packaging/CI gaps, then a TurboFieldfare-inspired feature pass (contract:
docs/addons/turbofieldfare-pass.md).Roadmap sweep — complete
v1.x candidates
cac1a9eshuttle_get_stats—55b4d89f4851d8SHUTTLE_DROP_NEWEST; overwrite-oldest rejected as unsound with rationale —60ec6c9v2 directions
windows-latestCI green —087b13a0bf303bGaps
d6e8b9a41d039422c98bbf07a9efa198ce5430b119TurboFieldfare pass — complete
110293dSHUTTLE_CREATE_ALIGNED_SPANS): old binaries reject aligned segments by design; worst-case fragmentation 8,183 B/message; bindings caught up to full ABI —2ec30dashuttle_create_file/open_file/unlink_file): EOWNERDEAD proven on a file-backed mapping with a negative control; durability an explicit non-goal —7d13543MADV_WILLNEEDprefetch (file-backed-gated) + read-onlyshuttle_peek_next(wrap case keepsread's single writing site; Rust pairsConsumer::peek_nextwithBorrowed::peek_next, pinned by a compile_fail doctest) —7dba295docs/EXPERIMENTS.md(E1–E5, including an honestly-recorded null result for prefetch on this host), full-branch README/CHANGELOG sweep, API.md v1.4 coherence + acquire_read doc corrected to implemented behavior —3a01927Final state
SHUTTLE_ABI_VERSIONstill 1🤖 Generated with Claude Code
https://claude.ai/code/session_01GGx1J3YCaJASQrZWq7XQFN