feat(fjall): implement AtomicAppend + StreamLister, prove export/import on disk (#220)#242
Merged
Merged
Conversation
…rt on disk (#220) The export/import pipeline (#145) was fully implemented but every test ran against InMemoryStore. The whole-chunk atomicity guarantee is meaningless until proven against a store with real cross-partition transactions (CLAUDE rule 1), which is why this card gates the 1.0 freeze. `nexus-fjall` now implements both capabilities the pipeline needs: - `AtomicAppend::atomic_append_many` commits N per-stream runs in ONE fjall `write_tx` spanning every partition (`streams`, `events`, `events_global`, `global`). Mirrors InMemoryStore's two-phase discipline — validate every run's head against a running projected head (so a non-injective route conflicts instead of concatenating a corrupt stream), then stage + assign GlobalSeq, then one atomic commit. Any conflict/overflow/encode failure drops the tx uncommitted: nothing lands across any partition. - `StreamLister::list_streams` lazily streams the `streams` partition's keys via a snapshot-pinned `fjall::Iter` (key-only, zero-copy `Slice -> Bytes`) — not InMemoryStore's eager `collect()`. `EventImporter` and `export_all` therefore work on fjall for free (blanket impls). The two read helpers gain bare `read_*_raw` cores so the atomic path maps errors honestly without inventing an impossible variant. Tests map onto the 4 cross-cutting categories: - in-crate white-box (need pub(crate) partition handles): the freeze-gating cross-partition rollback proof (every partition byte-identical after an aborted batch), non-injective route, defensive non-sequential run, all-or-nothing visibility + aborted-batch isolation under concurrent readers. - integration (tests/export_import_tests.rs): real list_streams + export_stream -> CBOR chunk -> temp file -> decode -> import into a fresh store, byte-equal modulo re-stamped global_seq; on-disk block corruption -> StreamOutcome::Corrupt vs malformed framing -> ChunkError::Malformed; whole-chunk corruption rollback. Feature wiring: `export`/`import` forward to nexus-store; a self dev-dependency unifies them into the default-feature `nix flake check` gate run (verified: the gated tests execute under `cargo test` with no --features), and a nexus-store `cbor` dev-dependency supplies the box codec. The nexus-store export/import contract is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #220.
The export/import pipeline (#145) was fully implemented but every test ran against
InMemoryStore. The whole-chunk atomicity guarantee is meaningless until proven against a store with real cross-partition transactions (CLAUDE rule 1) — which is why this card gates the 1.0 freeze. This PR implements the two missing fjall capabilities and proves the end-to-end pipeline on disk. Thenexus-storeexport/import contract is unchanged.Implementation
AtomicAppend for FjallStore—atomic_append_manycommits N per-stream runs in one fjallwrite_txspanning every partition (streams,events,events_global,global). MirrorsInMemoryStore's two-phase discipline: validate every run's head against a running projected head (so a non-injective route conflicts instead of concatenating a corrupt, non-monotonic stream) → stage + assignGlobalSeq→ one atomic commit. Any conflict / overflow / encode failure drops the tx uncommitted, so nothing lands across any partition. A secondwrite_txor a per-stream commit loop is exactly the bug this card exists to prevent.StreamLister for FjallStore—list_streamslazily streams thestreamspartition's keys via a snapshot-pinnedfjall::Iter(key-onlyGuard::key(), zero-copySlice → Bytes), notInMemoryStore's eagercollect().EventImporterandexport_alltherefore work on fjall for free (blanket impls).read_*_rawcores (returningFjallError, notAppendError) so the atomic path maps errors honestly without inventing an impossible variant;append's behavior is unchanged.Tests — the 4 cross-cutting categories (CLAUDE §7)
In-crate white-box (need the
pub(crate)partition handles):events_globalrows, stream version counter, global seq counter all unchanged).Conflict, no[1,2,1,2]corruption; defensive non-sequential run rejected; sequence/lifecycle (atomic + normal append share counters, survives reopen).Integration (
tests/export_import_tests.rs): reallist_streams+export_stream→ CBOR chunk → temp file on disk → reopen →decode_chunk→importinto a fresh fjall store, asserting byte-equality modulo re-stampedglobal_seq; on-disk block corruption →StreamOutcome::Corrupt(per-stream) /ImportError::Aborted(whole-chunk) vs malformed framing →ChunkError::Malformed.Gate coverage
export/importfeatures forward tonexus-store; a self dev-dependency unifies them into the default-featurenix flake checkrun (verified: the 14 gated unit tests + 6-test integration binary execute undercargo testwith no--features), and anexus-storecbordev-dependency supplies the box codec. Fullnix flake checkpasses.Notes
append(itsstreamskey is the raw id bytes). Pre-existing limitation, documented in CLAUDE.md, avoided in the fjall tests.🤖 Generated with Claude Code