Skip to content

feat: fjall adapter, store refactoring, serde codecs, repository builder#138

Merged
joeldsouzax merged 20 commits into
mainfrom
feat/fjall
Apr 9, 2026
Merged

feat: fjall adapter, store refactoring, serde codecs, repository builder#138
joeldsouzax merged 20 commits into
mainfrom
feat/fjall

Conversation

@joeldsouzax

@joeldsouzax joeldsouzax commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Major milestone: nexus goes from kernel-only to a full event sourcing stack with persistence.

  • nexus-fjall — Embedded LSM-tree event store adapter backed by fjall. Two partitions (streams metadata + events), LZ4 compression, bloom filters, atomic transactions, overflow-safe numeric ID allocation. Comprehensive test suite: state machine tests, property tests, adversarial resilience tests, snapshot wire format tests.
  • nexus-store refactoring — Modularized into codec/, envelope/, upcasting/, store/ directories. Introduced Store<S> Arc-wrapped shared handle. Separated RawEventStore (bytes) from typed facades (EventStore, ZeroCopyEventStore).
  • Serde + JSON codecs — Feature-gated SerdeCodec<F> with SerdeFormat trait abstraction. JsonCodec enabled via "json" feature. Zero-boilerplate event serialization.
  • RepositoryBuilder — Typestate builder replacing Store::repository(). Compile-time enforcement: can't build without codec (NeedsCodec is !Send). Auto-wires JsonCodec when "json" feature is enabled.
  • Unified event pipeline — Removed StreamId, migrated to Upcaster trait with compile-time validated #[nexus::transforms] macro, EventMorsel zero-copy pipeline.
  • Safety hardening — Version overflow protection, empty batch bypass fix, schema_version clamp, comprehensive CLAUDE.md rules (7 categories, 50+ rules).

Breaking Changes

  • StreamId removed — streams identified by any Display type via StreamLabel
  • Store::repository() / Store::zero_copy_repository() removed — use store.repository().codec(c).upcaster(u).build()
  • UpcasterChain removed — use #[nexus::transforms] macro or implement Upcaster trait directly
  • Event pipeline simplified: PendingEnvelope / PersistedEnvelope now in separate modules

Stats

  • 165 files changed, +18,699 / -5,046 lines
  • 19 commits

Test plan

  • CI passes (nix flake check: clippy, fmt, taplo, audit, deny, nextest, hakari)
  • All existing kernel tests pass
  • All nexus-store tests pass (event store, zero-copy, repository QA, upcaster, codec, envelope, builder, compile-fail)
  • All nexus-fjall tests pass (state machine, property, resilience, wire format snapshots)
  • Serde codec tests pass with feature gates
  • Examples updated and compile (inmemory, store-inmemory, store-and-kernel)

🤖 Generated with Claude Code

joeldsouzax and others added 20 commits April 4, 2026 17:04
@joeldsouzax @claude
feat(fjall): scaffold nexus-fjall crate
bc30cbd 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat(fjall): encoding module with TDD for key/value byte layouts
d492c54 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat(fjall): error type, FjallStream, and FjallStore with builder
90447d9 
Implement Tasks 3-5 for the nexus-fjall crate:

- FjallError enum with Io, CorruptValue, and CorruptMeta variants
  satisfying Error + Send + Sync + 'static
- FjallStream implementing EventStream trait as a lending cursor
  over eagerly-loaded fjall range scan results, with debug_assertions
  monotonicity checking
- FjallStore struct holding TxKeyspace, two TxPartitionHandles
  (streams + events), and AtomicU64 counter for stream ID allocation
- FjallStoreBuilder with configurable partition options and recovery
  of next_stream_id on reopen

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat(fjall): RawEventStore impl with append and read_stream
b269ba3 
Implement the RawEventStore trait for FjallStore with atomic
append (transactional version check + insert + commit) and
range-scan-based read_stream. Remove dead_code allows from
encoding functions and FjallStore struct now that they are used.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat(fjall): conformance tests, persistence, and pub API verification
366a991 
Add 6 conformance tests: data persistence across reopen, stream ID
counter recovery, version tracking, large batch (100 events), schema
version round-trip, and empty batch no-op. All 33 tests pass with
clippy --deny warnings and fmt clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
fix(fjall): address code review findings (tempdir, try_new, empty batch)
91a388e 
- C1: temp_store() now returns (FjallStore, TempDir) so TempDir lives
  until each test ends; all callers destructure as (store, _dir)
- S2: Use PersistedEnvelope::try_new() instead of new() in FjallStream
  to avoid panic on corrupt data with schema_version == 0
- S4: Early return in append() when envelopes slice is empty to avoid
  creating phantom stream entries
- I3: Added comment explaining why Relaxed ordering is safe for
  fetch_add on next_stream_id

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
perf(fjall): add bloom filter, point-read optimization, and LZ4 defaults
8735914 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
test(fjall): NASA-grade adversarial test suite with property-based te…
d90841c 
…sting

42 tests across 10 attack categories: encoding roundtrips, append-read
integrity, evil stream IDs, version boundary arithmetic, concurrency
races, persistence/recovery, stress/scale, schema version edges,
model-based shadow store, and degenerate cases.

BUG FOUND: empty string stream ID causes panic in fjall's lsm-tree
(key may not be empty). FjallStore::append should validate non-empty
stream_id before passing to fjall. Documented in test
attack_empty_string_stream_id_panics_bug.

Also makes encoding module items pub (from pub(crate)) with proper
#[must_use] and error docs so integration tests can exercise them
directly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
fix(fjall)!: prevent empty stream_id panic and numeric ID overflow
4f413af 
Two bugs fixed:
- Empty stream_id caused panic in fjall's lsm-tree ("key may not be
  empty"). Now returns FjallError::EmptyStreamId at the boundary.
- Numeric stream ID counter used fetch_add which wraps silently on
  u64::MAX overflow, causing two different stream_id strings to share
  the same key space (silent data corruption). Now uses checked
  arithmetic and returns FjallError::StreamIdExhausted.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
docs: StreamId newtype design — validated, type-safe stream identifiers
4c42427 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat(nexus): add StreamId newtype with validated construction
fa74bc5 
Introduce a StreamId newtype that enforces invariants at construction
time: aggregate names must be ASCII alphanumeric + underscore, IDs
must be non-empty, and the combined string must be null-free and within
512 bytes. A separate from_persisted constructor allows store adapters
to load historical stream IDs with relaxed format checks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat(store)!: PendingEnvelope and traits take StreamId instead of &str
a84934d 
BREAKING CHANGE: RawEventStore, Repository, PendingEnvelope now use
StreamId for compile-time stream ID validity.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
refactor(fjall): remove dead EmptyStreamId variant
c390911 
StreamId newtype makes empty stream IDs impossible at the type level.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
test(fjall): military-grade DST, overflow, and concurrency attack suite
035af55 
36 penetration tests across 18 attack categories: version arithmetic
overflow proof, dual-instance corruption probe, deterministic simulation
with shadow model, snapshot isolation, crash simulation via mem::forget,
key boundary conditions, recovery torture (100 streams x 5 reopens),
concurrent append storm (50 tasks), encoding boundary attacks, stream ID
exhaustion, and more.

Bugs documented (not fixed):
1. store.rs:109 — unchecked u64 addition `current_version + 1 + i_u64`
   overflows when current_version is near u64::MAX (panic in debug,
   silent wrap in release)
2. store.rs:61 — empty batch early return bypasses version validation,
   allowing append with wrong expected_version to succeed silently
3. schema_version=0 accepted on write but rejected on read (data written
   that can never be read back)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
fix!: version overflow, empty batch bypass, schema_version clamp, rec…
f80950b 
…ursive asserts

Bug 1 (CRITICAL): Version arithmetic overflow in sequential validation.
  current_version + 1 + i uses unchecked addition — panics in debug,
  wraps silently in release. Fixed with checked_add in both InMemoryStore
  and FjallStore.

Bug 2 (MEDIUM): Empty batch bypasses version check in FjallStore.
  append(nonexistent, version_999, &[]) succeeded despite version
  mismatch. Fixed: version check now runs before empty-batch early return.

Bug 3 (MEDIUM): schema_version=0 write-read asymmetry.
  PendingEnvelope builder accepted 0, but PersistedEnvelope::try_new
  rejects 0 on read — creating unreadable data. Fixed: builder clamps
  0 to 1.

Bug 4 (CRITICAL): Recursive debug_assert in encoding functions.
  encode_event_key called decode_event_key in a postcondition, which
  called encode_event_key — infinite mutual recursion causing stack
  overflow in all debug-mode tests. Removed all mutually-recursive
  postconditions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat!: unified event pipeline — remove StreamId, migrate to Upcaster …
0379210 
…trait, simplify API

Major architectural refactoring across all crates:

- Remove StreamId newtype from kernel; streams identified by generic Id
- Replace TransformChain/SchemaTransform with single Upcaster trait
- Add EventMorsel type for zero-copy event transformation
- Add StreamLabel for type-safe stream identification in store layer
- Add #[nexus::transforms] proc macro for compile-time upcaster generation
- Simplify FjallStore (~400 lines removed), remove redundant validation
- Replace exhaustive test suites (military/nasa_break_everything) with
  focused property tests, resilience tests, and wire format snapshots
- Update all examples, compile-fail tests, and benchmarks for new API
- Include .snap files in nix source filter for insta snapshot tests
- Update CLAUDE.md with comprehensive coding rules from audit

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
refactor(store)!: introduce Store<S> wrapper, modularize nexus-store
5f0ecc0 
- Add Store<S> as Arc-wrapped shared handle to any RawEventStore
- Store::repository(codec, upcaster) creates typed EventStore facades
- Store::zero_copy_repository(codec, upcaster) for borrowing codecs
- Split EventStore and ZeroCopyEventStore into separate modules
- Reorganize nexus-store from flat files into codec/, envelope/,
  store/, and upcasting/ directories
- Migrate all tests and examples to new Store::repository() API
- Update CLAUDE.md architecture documentation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat(store): add serde + json feature-gated codecs
8241240 
Adds `serde` and `json` features to nexus-store so users get ready-made
codecs without hand-implementing `Codec<E>`:

- `serde` feature: `SerdeFormat` trait + `SerdeCodec<F>` generic codec
- `json` feature (implies serde): `Json` format + `JsonCodec` alias

Users derive Serialize/Deserialize on their event enum and use
`JsonCodec::default()` — zero boilerplate codec implementation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
feat(store)!: replace repository() with typestate RepositoryBuilder
3999033 
Replace `Store::repository(codec, upcaster)` and
`Store::zero_copy_repository(codec, upcaster)` with a fluent builder:

  store.repository().codec(c).upcaster(u).build()
  store.repository().codec(c).build_zero_copy()

With the `json` feature, JsonCodec is the default — zero-config usage:

  store.repository().build()

NeedsCodec (!Send) prevents calling build() without a codec configured.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax @claude
chore: update local Claude Code permissions
86ff001 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@joeldsouzax joeldsouzax merged commit c5b3eab into main Apr 9, 2026
2 checks passed
@joeldsouzax joeldsouzax deleted the feat/fjall branch April 9, 2026 15:30
@joeldsouzax joeldsouzax mentioned this pull request Apr 10, 2026
Closed
5 tasks
@github-actions github-actions Bot mentioned this pull request Jun 29, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@joeldsouzax