refactor(store, fjall, framework)!: stream + codec collapse + wire alignment — PR2 of bytes-envelope refactor#183
Merged
Conversation
…e refactor Foundation for the codec/stream collapse: - aligned-vec = "=0.6.4" pinned (0.x, no semver between minors) - bytemuck = "1" optional, gated by feature `bytemuck` - rkyv = "0.8" optional, gated by feature `rkyv` - futures-core promoted from optional `futures-bridge` to required - futures (workspace) promoted from dev-dep to regular dep (need StreamExt) - futures-bridge feature kept as no-op until Task 6 deletes its cfg refs (see deviations log) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ith 16-byte payload alignment Bundles Tasks 2 + 3 of PR2 (bytes-envelope refactor): - New `nexus_store::wire` module is the single source of truth for the on-disk row layout. `build_row` produces an `AVec<u8, ConstAlign<16>>` wrapped in `bytes::Bytes::from_owner`, ensuring the payload pointer is 16-byte aligned in every adapter's stored value. - Wire layout: `[u64 LE global_seq][u32 LE schema_version][u16 LE event_type_len][u32 LE meta_len][event_type][metadata?][padding] [payload]`. Padding rounds the payload offset to a 16-byte boundary. `decode_row` is the inverse. - `FjallStore::append` calls `wire::build_row` directly; the existing `encode_event_value` / `decode_event_value` in fjall::encoding become thin wrappers that preserve the adapter-local test/bench API. - `InMemoryStore::StoredRow` drops its separate global_seq / schema_version / per-field range fields — they all live in `value: Bytes` + `offsets: RowOffsets` now. The cursor parses global_seq + schema_version from the wire bytes on yield. New `InMemoryStoreError::CorruptGlobalSeq` covers the GlobalSeq=0 case. - Re-enables `zero_copy_save_and_load_roundtrip` and `zero_copy_multi_save_load` from PR1 (#182). rkyv, flatbuffers, and repr(C) POD borrowing decoders are unblocked. - Proptests verify payload alignment across arbitrary inputs and that build_row → decode_row round-trips header fields and offsets. Wire signature deviates from the plan's spec to include `global_seq` + `schema_version` — see docs/plans/2026-05-27-bytes-envelope-deviations.md for the rationale (the plan's signature would have lost both fields on fjall's read path, and prepending a fjall-local header would break the alignment invariant). Snapshot tests for wire format updated to reflect the new padding bytes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end Bytes flow from codec to wire. SerdeCodec adapts with a single Bytes::from on its serializer output (zero-copy ownership transfer of the Vec backing). Test fixture codecs and examples updated mechanically — Bytes::from(vec) for owned buffers, Bytes::copy_from_slice for slice references, Bytes::from_static for literal arrays. CodecSnapshotStore::commit copies Bytes->Vec for the underlying SnapshotStore<Vec<u8>, P> adapter (snapshot writes are rare relative to the read path, so the extra allocation is acceptable). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two-trait split was a workaround for the borrowed-cursor lifetime cliff. PR1's owned-Bytes envelope removes the cliff: the same operation no longer needs two trait shapes. One trait, GAT-parameterized output: type Output<'a>; fn decode<'a>(&'a self, env: &'a PersistedEnvelope) -> Result<...>; Serde owning codecs: Output<'a> = E. Borrowing codecs (rkyv, bytemuck, raw &[u8]): Output<'a> = &'a E. decode() takes the whole PersistedEnvelope rather than two extracted arguments — the envelope already carries every field a codec needs. EventStore / ZeroCopyEventStore facades constrain Output<'a> shape via HRTB equality (for<'a> C: Decode<E, Output<'a> = E or &'a E>). Three follow-ons the plan did not call out (see deviation log): - StoreError::Wire(WireError) + CodecSnapshotStoreError::Wire variants surface wire-format failures from upcaster-driven envelope synthesis and snapshot byte-wrapping. Practically unreachable for in-budget inputs but typed for correctness. - PersistedEnvelope::for_decode(event_type, payload) helper synthesizes a 16-byte-payload-aligned envelope for callers without a real one: upcaster post-transform, snapshot decode, and codec round-trip tests. - repository_qa_tests.rs split DeltaBorrowingCodec into two structs (DeltaBorrowingCodec + DeltaOwningCodec) — coherence forbids two Decode<E> impls on one type with different Output<'a>. Also drive-by clippy hygiene on wire.rs test module + stream_alloc_tests + metadata_roundtrip_tests (shadow / as / is_multiple_of). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ker over futures::Stream
Bundle of Tasks 6-11 of PR2's bytes-envelope refactor (see deviation log:
intermediate states would have left workspace uncompilable, violating the
never-skip-flake-check rule).
• nexus-store::stream — delete cursor.rs / combinators.rs / progress.rs /
owned.rs / mod.rs (~1370 lines). Replace with a 50-line marker trait
in stream.rs over futures::Stream<Item = Result<PersistedEnvelope, _>>.
Combinators come from futures::StreamExt / TryStreamExt for free.
• Drop M = () generic from RawEventStore, Subscription, SubscriptionBackend,
and the Arc<T> blanket. RawEventStore::Stream<'a> GAT → concrete owned
Stream associated type.
• nexus-fjall: FjallStream and FjallSubscriptionStream rewritten as impl
futures::Stream. FjallStream eagerly loads rows into a VecDeque at
construction; poll_next is pure sync. FjallSubscriptionStream boxes an
unfold-driven inner stream that cycles between yield, refill via
FjallStore::read_stream, and wait on FjallStore::notify.
• nexus-store::testing: InMemoryStream and InMemorySubscriptionStream
mirror the fjall shape.
• nexus-store::repository: EventStore::load and ZeroCopyEventStore::load
drive the stream via futures::TryStreamExt::try_fold + async move
closures. BaseEventStream::to_envelope indirection gone. The upcaster
F is wrapped in Arc<F> so per-iteration clones into the FnMut closure
work.
• nexus-framework::projection: Projection::run replaces the bespoke
try_fold_async_until combinator with a tokio::select! between
stream.next() and the shutdown future. tokio promoted to
[dependencies] (features = ["macros"]).
• futures crate added as explicit dep on nexus-fjall, nexus-framework,
nexus-store-testing, and the two examples. futures-bridge no-op
feature removed.
• Test cleanup: deleted stream_tests.rs, combinator_tests.rs,
futures_bridge_tests.rs, and the compile_fail stream_not_send fixture.
• Test fixtures (VecStream, ProbeStream, OwnedFjallStream, benchmark
fixtures) migrated from impl BaseEventStream + EventStream<Item<'a>>
to impl futures::Stream<Item = Result<PersistedEnvelope, _>>.
• Test body shape flip: Result<Option<E>, _> → Option<Result<E, _>>.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt tests
Closes out Tasks 12, 13, and 15 of PR2's bytes-envelope refactor.
• BytemuckCodec (feature `bytemuck`) — zero-copy codec for POD types.
Output<'a> = &'a E, borrowing directly from the envelope's payload
bytes (16-byte aligned by wire-format invariant). Wraps the upstream
bytemuck::PodCastError in a local BytemuckError because the upstream
type is no_std and doesn't impl std::error::Error. Round-trip test
asserts pointer equality between env.payload() and the decoded &E
(proves zero-copy).
• RkyvCodec (feature `rkyv`) — zero-copy codec via rkyv 0.8.
Output<'a> = &'a <E as Archive>::Archived. Encode goes through
rkyv::to_bytes_in(value, AlignedVec::new()); decode uses
rkyv::access for validation + cast. The AlignedVec → Vec<u8> →
Bytes conversion loses alignment, but wire::build_row re-aligns
the payload on write so the envelope's payload regains alignment.
Bound chain matches rkyv 0.8.16's HighSerializer/HighValidator
types verbatim.
• bytemuck = "1" workspace dep gains the `derive` feature so
`#[derive(bytemuck::AnyBitPattern, bytemuck::NoUninit)]` works
in the codec round-trip test.
• tests/wire_alignment_tests.rs — four cross-cutting tests, one per
CLAUDE.md-mandated category:
1. Sequence: 20 appends on one stream; every yielded payload pointer
is 16-byte-aligned.
2. Lifecycle: clone an `Arc<InMemoryStore>` and read through a
fresh cursor; payload still aligned.
3. Defensive boundary: 11 event_type lengths spanning the
{0, 1, 6, 13, 14, 15, 16, 17, 30, 100, 1024} positions around
the 16-byte header-extent seam — verifies the padding logic.
4. Linearizability: concurrent writer (50 appends) and reader
(30 reread loops) on multi_thread tokio runtime; every read
payload is aligned.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
joeldsouzax
added a commit
that referenced
this pull request
Jun 4, 2026
Documentation closeout for the 2026-05-27 bytes-envelope refactor. Brings prose in line with what shipped in PR1 (#182) and PR2 (#183): - README.md: rewrite "Zero-copy read path" to describe the collapsed Decode<E> trait with Output<'a> GAT (replaces the deleted BorrowingCodec + GAT lending iterator story); add bytemuck / rkyv to the codec feature list; document the 16-byte payload-alignment wire-format invariant; list nexus-framework as a fifth crate. - Crate-level //! rustdoc on nexus-store/src/lib.rs, nexus-store/src/codec.rs, nexus-fjall/src/lib.rs, nexus-framework/src/lib.rs. Explains the *why* behind each subsystem — the lifetime-cliff that justified the Decode/BorrowingDecode split (and why the owned-Bytes envelope removed it), fjall's load-bearing `bytes_1` feature, the runtime-agnostic boundary nexus-framework enforces. - CLAUDE.md: rewrite the codec.rs, envelope.rs, store.rs, repository.rs, state.rs bullets in the nexus-store section; add a wire.rs bullet (the module didn't exist pre-refactor); add a new stream.rs bullet replacing the old GAT-lending description. Refresh the nexus-fjall section to describe FjallStream / FjallSubscriptionStream as concrete `impl futures::Stream` types, mention the `bytes_1` feature, point at wire::build_row as the canonical row builder. Refresh the nexus-framework section to describe the tokio::select! / pin! event loop that replaced the bespoke try_fold_async_until combinator. Enrich the store-inmemory example description to call out the futures::TryStreamExt::try_fold substrate path. Also (scope expansion authorized in PR3 planning): - Fix latent flaky proptest attack_store_error_conflict_preserves_info. The original strategy was `expected in 0..10000u64`, but Version::new(0) returns None (Version is NonZeroU64), so the error format rendered "None" instead of "0" and the .contains("0") assertion failed on ~40% of fresh seeds. Replaced both ranges with prop_oneof! strategies that include None, Version::new(1), Version::new(u64::MAX), and an interior range — satisfies the project's exhaustive-boundary-coverage rule. Assertions now check the Debug rep of each Option<Version> verbatim, which is the actual semantic invariant. Captured the failing seed in the proptest-regressions file so future drift is caught immediately. - Delete tracked stray file with a sed-fragment name that landed on main via a prior session's regex mishap. Logged the test-fix scope expansion in docs/plans/2026-05-27-bytes-envelope-deviations.md. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR2 of the bytes-envelope refactor. Builds on #182 (PR1, owned-Bytes envelopes).
This PR does three structurally connected things in one merge window because they touch overlapping crate surface (
nexus-store::codec+nexus-store::envelope+nexus-store::stream+nexus-fjall+nexus-store::testing+nexus-framework):Stream trait collapse
~1,370 lines of GAT-lending stream machinery deleted (
cursor.rs,combinators.rs,progress.rs,owned.rs,mod.rs). Replaced with a 50-line marker trait instream.rs:Auto-impl'd for every matching
futures::Stream. Combinators come fromfutures::StreamExt/TryStreamExtfor free.RawEventStore::Stream<'a>GAT → concrete ownedStreamassociated type.M = ()generic dropped fromRawEventStore,Subscription,SubscriptionBackend(it was always()in practice).FjallStreamandFjallSubscriptionStreamrewritten asimpl futures::Streamwithunfoldfor the subscription's notify/refill cycle. Same forInMemoryStreamandInMemorySubscriptionStream.Codec trait collapse
Decode<E>+BorrowingDecode<E>merged into oneDecode<E>with atype Output<'a>GAT:Output<'a> = E.Output<'a> = &'a <E as Archive>::Archived.Output<'a> = &'a E.Encode::encodenow returnsbytes::Bytesinstead ofVec<u8>for end-to-end zero-copy from codec to wire.Wire-format payload-alignment invariant
nexus-store::wire::build_row(added in commit 31a5c34 of this branch) pads every row's payload to a 16-byte boundary inside anAVec-backedBytes. Bothnexus-fjall::FjallStore::appendandnexus-store::testing::InMemoryStore::appendgo through the same builder.This unblocks
BytemuckCodec(zero-copyalign_to::<E>) andRkyvCodec(zero-copyaccess::<E::Archived>) — both ship in this PR asnexus-storefeature flags.Two new codecs
bytemuck) — zero-copy for POD types.Output<'a> = &'a Eborrowing fromenv.payload(). LocalBytemuckErrorwrapper because upstreamPodCastErrorisno_stdand doesn't implementstd::error::Error. Round-trip test asserts pointer equality betweenenv.payload()and the decoded&E.rkyv) — zero-copy via rkyv 0.8.Output<'a> = &'a <E as Archive>::Archived. Encode viato_bytes_in(value, AlignedVec::new()); decode viaaccess.AlignedVec→Vec<u8>→Bytesloses alignment, butwire::build_rowre-aligns on write so the envelope's payload regains it.Migration
BorrowingDecode<E>→Decode<E>withtype Output<'a> = &'a E. Mechanical for codecs that match one of the two shapes; coherence-conflict only fires if a single codec previously implemented bothDecodeandBorrowingDecodefor the sameE(rare).Decode::decode(name, payload)→Decode::decode(env)— codecs reach forenv.event_type()andenv.payload()themselves.Encode::encodereturn typeVec<u8>→Bytes(mechanical viaBytes::from(vec)— zero-copy ownership transfer).EventStreamExt::try_fold→futures::TryStreamExt::try_fold(same shape; closure body must now beasync move).BaseEventStream::to_envelopeindirection gone — cursors yield envelopes directly.Two new StoreError variants
StoreError::Wire(WireError)— reachable only from upcaster-drivenload_withpaths where a fresh aligned envelope is synthesized from the transformed morsel viawire::build_row. Practically unreachable for in-budget inputs.CodecSnapshotStoreError::Wire(WireError)— the same for snapshot decode (wraps raw snapshot bytes in a synthetic envelope to call the newDecode::decode(env)signature).Helper
PersistedEnvelope::for_decode(event_type, payload) -> Result<Self, WireError>wraps raw bytes in a synthetic 16-byte-aligned envelope for callers without a real one: upcaster post-transform, snapshot decode, codec round-trip tests.Test plan
nix flake check— clippy strict, fmt, taplo, audit, deny, hakari, nextest. Pre-commit hook enforced on every commit in this branch.wire_alignment_tests.rscross-cutting tests (sequence, lifecycle, defensive boundary, linearizability) assert payload pointer % 16 == 0 across every yielded envelope.wire::testsproptests verify payload alignment across arbitrary inputs.zero_copy_save_and_load_roundtrip+zero_copy_multi_save_loadPASS (alignment regression from PR1 is now fixed).BytemuckCodecround-trip + pointer-equality zero-copy assertion + wrong-size rejection.RkyvCodecround-trip on archived fields.nexus-fjall::FjallStoreconformance vianexus-store-testingharness.Companion docs
docs/plans/2026-05-27-bytes-envelope-design.mddocs/plans/2026-05-28-bytes-envelope-pr2-implementation.mddocs/plans/2026-05-27-bytes-envelope-deviations.md— every divergence from the plan logged with reason + impact, including the Task 6-11 bundling rationale.Why now (driving trigger)
Postgres adapter realism. sqlx's
BYTEAdecode path has noBytesimpl and the row buffer'sBytesfield ispub(crate)— a borrowed-row-bufferPersistedEnvelope<'a>can never serve a sqlx-style driver cleanly. Once postgres is on the roadmap, GAT lending streams structurally cannot serve all targets. The bytes-envelope pattern unifies fjall (zero-copy underbytes_1), postgres (unavoidable one alloc on decode), and any future driver under one envelope shape.🤖 Generated with Claude Code