Skip to content

feat(fjall): implement AtomicAppend + StreamLister, prove export/import on disk (#220)#242

Merged
joeldsouzax merged 1 commit into
mainfrom
feat/220-fjall-export-import-coverage
Jun 28, 2026
Merged

feat(fjall): implement AtomicAppend + StreamLister, prove export/import on disk (#220)#242
joeldsouzax merged 1 commit into
mainfrom
feat/220-fjall-export-import-coverage

Conversation

@joeldsouzax

Copy link
Copy Markdown
Contributor

Closes #220.

The export/import pipeline (#145) was fully implemented but every test ran against InMemoryStore. The whole-chunk atomicity guarantee is meaningless until proven against a store with real cross-partition transactions (CLAUDE rule 1) — which is why this card gates the 1.0 freeze. This PR implements the two missing fjall capabilities and proves the end-to-end pipeline on disk. The nexus-store export/import contract is unchanged.

Implementation

  • AtomicAppend for FjallStoreatomic_append_many commits N per-stream runs in one fjall write_tx spanning every partition (streams, events, events_global, global). Mirrors InMemoryStore's two-phase discipline: validate every run's head against a running projected head (so a non-injective route conflicts instead of concatenating a corrupt, non-monotonic stream) → stage + assign GlobalSeq → one atomic commit. Any conflict / overflow / encode failure drops the tx uncommitted, so nothing lands across any partition. A second write_tx or a per-stream commit loop is exactly the bug this card exists to prevent.
  • StreamLister for FjallStorelist_streams lazily streams the streams partition's keys via a snapshot-pinned fjall::Iter (key-only Guard::key(), zero-copy Slice → Bytes), not InMemoryStore's eager collect().
  • EventImporter and export_all therefore work on fjall for free (blanket impls).
  • The two read helpers gain bare read_*_raw cores (returning FjallError, not AppendError) so the atomic path maps errors honestly without inventing an impossible variant; append's behavior is unchanged.

Tests — the 4 cross-cutting categories (CLAUDE §7)

In-crate white-box (need the pub(crate) partition handles):

  • Freeze gate — a mid-batch conflict leaves every partition byte-identical to the pre-import snapshot (event rows, events_global rows, stream version counter, global seq counter all unchanged).
  • non-injective route → Conflict, no [1,2,1,2] corruption; defensive non-sequential run rejected; sequence/lifecycle (atomic + normal append share counters, survives reopen).
  • Linearizability: clean multi-event batch is visible all-or-nothing to a concurrent reader; an aborted batch never leaks its writes to a concurrent reader.

Integration (tests/export_import_tests.rs): real list_streams + export_stream → CBOR chunk → temp file on disk → reopen → decode_chunkimport into a fresh fjall store, asserting byte-equality modulo re-stamped global_seq; on-disk block corruption → StreamOutcome::Corrupt (per-stream) / ImportError::Aborted (whole-chunk) vs malformed framing → ChunkError::Malformed.

Gate coverage

export/import features forward to nexus-store; a self dev-dependency unifies them into the default-feature nix flake check run (verified: the 14 gated unit tests + 6-test integration binary execute under cargo test with no --features), and a nexus-store cbor dev-dependency supplies the box codec. Full nix flake check passes.

Notes

🤖 Generated with Claude Code

…rt on disk (#220)

The export/import pipeline (#145) was fully implemented but every test ran
against InMemoryStore. The whole-chunk atomicity guarantee is meaningless until
proven against a store with real cross-partition transactions (CLAUDE rule 1),
which is why this card gates the 1.0 freeze.

`nexus-fjall` now implements both capabilities the pipeline needs:

- `AtomicAppend::atomic_append_many` commits N per-stream runs in ONE fjall
  `write_tx` spanning every partition (`streams`, `events`, `events_global`,
  `global`). Mirrors InMemoryStore's two-phase discipline — validate every
  run's head against a running projected head (so a non-injective route
  conflicts instead of concatenating a corrupt stream), then stage + assign
  GlobalSeq, then one atomic commit. Any conflict/overflow/encode failure drops
  the tx uncommitted: nothing lands across any partition.
- `StreamLister::list_streams` lazily streams the `streams` partition's keys
  via a snapshot-pinned `fjall::Iter` (key-only, zero-copy `Slice -> Bytes`) —
  not InMemoryStore's eager `collect()`.

`EventImporter` and `export_all` therefore work on fjall for free (blanket
impls). The two read helpers gain bare `read_*_raw` cores so the atomic path
maps errors honestly without inventing an impossible variant.

Tests map onto the 4 cross-cutting categories:
- in-crate white-box (need pub(crate) partition handles): the freeze-gating
  cross-partition rollback proof (every partition byte-identical after an
  aborted batch), non-injective route, defensive non-sequential run,
  all-or-nothing visibility + aborted-batch isolation under concurrent readers.
- integration (tests/export_import_tests.rs): real list_streams + export_stream
  -> CBOR chunk -> temp file -> decode -> import into a fresh store, byte-equal
  modulo re-stamped global_seq; on-disk block corruption -> StreamOutcome::Corrupt
  vs malformed framing -> ChunkError::Malformed; whole-chunk corruption rollback.

Feature wiring: `export`/`import` forward to nexus-store; a self dev-dependency
unifies them into the default-feature `nix flake check` gate run (verified: the
gated tests execute under `cargo test` with no --features), and a nexus-store
`cbor` dev-dependency supplies the box codec.

The nexus-store export/import contract is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@joeldsouzax joeldsouzax merged commit 8e67215 into main Jun 28, 2026
2 checks passed
@joeldsouzax joeldsouzax deleted the feat/220-fjall-export-import-coverage branch June 28, 2026 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

export/import: prove the pipeline on a persistent adapter (fjall AtomicAppend + StreamLister + end-to-end coverage)

1 participant