MVP History

Felix MVP To-Do List

The MVP described here was completed, and Felix has moved well past it. This file is kept as the historical checklist and for the design sketches and answered open questions further down, which are still the reasoning behind the current storage layer.

It is not the place to look for current status. Two places are:

• The status table, kept current per capability, and the authority when anything disagrees with it.
• GitHub issues and milestones for what is being worked on.

What the MVP Achieved

A single-node broker that accepts QUIC connections, supports publish/subscribe over a framed v1 protocol, and provides an in-memory TTL cache — focused on correctness and operability rather than durability, clustering, or advanced observability. Durability, clustering, replication, failover, the log-backed cache, and consumer groups all landed afterwards.

---

Build + repo health (Done)

• cargo build --workspace succeeds cleanly
• cargo test --workspace exists and runs
• Taskfile.yml targets: build, test, fmt, lint, demo
• Add CI workflow: fmt, clippy, test
• Add baseline deps management: cargo-deny + deny.toml

Wire protocol v1 (felix-wire)

• Versioned frame header (magic/version/flags/len)
• Add message type in header (type field) for faster dispatch
• Message types: Publish, Subscribe, Event, CachePut, CacheGet, CacheValue, Ok, Error
• Encode/decode message payloads with frames
• Define v1 wire spec in docs/protocol.md
• Add test vectors in crates/felix-wire/tests/vectors/
• Add conformance runner tool (felix-conformance)
• Add fuzz tests for frame + message decoding (storage decoding is covered: see crates/felix-storage/fuzz/)
• Add compatibility notes (reserved fields for future encryption/compression)

QUIC transport (felix-transport)

• QUIC server endpoint wrapper (quinn)
• QUIC client endpoint wrapper
• Connection info + stream helpers (bi/uni)
• Graceful shutdown hooks (drain connections on SIGTERM) — see Graceful Shutdown. Readiness flip, accept-loop cancellation, and a bounded per-connection drain (TaskTracker grace window, then the connection is closed) are done. Cancelling each subsystem individually was deliberately not the design: publish and subscribe streams are long-lived by construction, so waiting on them would hang exactly as long as waiting on the connection. What is still untested is a real signal: run_with_shutdown is driven by a synthetic future in tests, and nothing sends an actual SIGTERM to a running process.
• Backpressure defaults (caps per connection/subscription)

Broker core (felix-broker)

• In-memory stream registry (explicit registration)
• Fanout to N subscribers
• Cursor model (tail-only is fine for MVP)
• Append-only in-memory log per stream (ring buffer)
• Cache: Put/Get with TTL expiry
• Cache eviction policy placeholder (size cap + LRU later)

Broker service (services/broker)

• Wire QUIC transport to broker protocol handler
• Env var for QUIC bind address (FELIX_QUIC_BIND)
• Health endpoints: /live, /ready
• Metrics endpoint (Prometheus)

Demo + examples (Done)

• QUIC broker demo using framed messages
• Demo for cache put/get over QUIC (or document separate cache API)
• Demo for testing latency with a comprehensive-ish matrix

MVP Definition of Done

• Single broker accepts QUIC connections
• Client can publish to a named stream over QUIC
• Subscribers receive stream events over QUIC
• Cache put/get available over QUIC
• Latency target: p999 <= 1 ms for small payloads on localhost baseline — met with margin: 214–251 µs at fanout 1 and 340–483 µs at fanout 10. See Benchmarks.
• Basic metrics exist and show throughput/latency
• Unit tests cover wire encode/decode and broker fanout behavior

Post-MVP (out of scope for the minimal MVP; since delivered)

• Queue semantics — consumer groups with poll, acknowledge, hand-back, visibility-timeout redelivery, an attempt bound, and dead letters.
• Distributed cache — though not by Raft. A key routes to one owner and the shard is replicated by leader leases and log shipping; per-shard Raft was considered and rejected in replication-design.md.
• Multi-node clustering and replication
• Design sharding/replication/quorum to move beyond single-node — settled in replication-design.md. Raft remains the intended answer for control-plane metadata and not for replicating records.

Data scalability with sharding

• Streams should be defined with shard count
• Caches should be defined with shard count (Cache::shards)
• Ops should be directed to the correct shard leader — a publish or cache operation for a shard this broker does not own is forwarded to the owner.
• Leader election should be managed — the control plane assigns, and a lease fences the assignment.
• Leader failover should be handled — by promotion of a replica that holds the log, not by Raft.

Data durability and persistence

• Add durable backend for control plane — Postgres, with sqlx::migrate!.
• Add durable backend for data plane — segmented, checksummed, crash-safe log behind StreamMetadata::durable. See Durable Storage.
• Handle crash/recover of broker data-plane nodes (torn-tail repair on startup; loud failure on interior corruption). Control-plane recovery and re-sync from a new leader remain open.
• Enforce retention — sealed segments are deleted on age or size, base_offset advances, and offsets below it report Trimmed / CursorTooOld. Off by default. See Durable Storage.
• Define requirements for tiered storage (hot/cold path, LCU?) — tracked as #172
• Implement tiered storage primitives. TieredStore and friends are declared in crates/felix-storage/src/tiered.rs and nothing implements them; see #172 for what M1's sealed segments already set up for it.

Durable storage sketch

• Split durable storage into Index and Data files: Index holds metadata (key + pointer/status), data file holds payload bytes.
• Keep keys (index entries) in memory as a write-through cache; writes first grow the data file, mark the index entry “in use,” then write payload bytes so that the system never reports data as durable until bytes hit disk.
• The append-only data file with in-memory index mimics a simple log (LSM-style without levels) and gives us a predictable replay order for fanout and replication.

Open questions

1. Do we need checksums/hashes to detect silent data corruption during reads or after reboots? Answered: yes. Every record carries a CRC-32 over its header and payload, verified on every read and during recovery. See the format spec.
2. How should data file segmentation and garbage collection work to honor per-segment size caps? What is a sane segment cap? 512MB? Answered: segments roll at segment_size_bytes, default 256 MiB — chosen so a full scan of the active segment at startup stays under a second. Collection is whole-segment retention by age or size, off by default; see Durable Storage.
3. What is the delete policy? Should we retain the last N versions per key for rollbacks, or can we drop them immediately?
4. What crash recovery guarantees do we need? We can’t mark an entry as committed until the payload bytes are actually persisted. Answered: three policies with explicit windows — OnCommit acknowledges only after a device flush, Periodic bounds loss by its interval, None makes no promise beyond the page cache. See Durable Storage.

Performance optimization

• Figure out how to handle backpressure — bounded queues at every stage with an explicit overflow policy; defaults validated under sustained overload.
• Measure P999 tail latency and throughput
• Identify optimizations and minspec clustering for optimization

Kubernetes deployment + scale units

• Define scale unit boundaries (per-tenant vs per-namespace vs per-cache/stream)
• Create Helm chart or Kustomize manifests for control plane + broker
• Add ConfigMap/Secret wiring (QUIC bind, control plane URL, auth keys)
• Implement readiness/liveness endpoints used by probes
• Expose service types (ClusterIP + optional LoadBalancer) for broker QUIC
• Add horizontal scaling plan (replicas + shard assignment)
• Add PodDisruptionBudget and resource requests/limits
• Document deployment topology and upgrade/rollback steps

---

Generated from docs/todos.md in the repository. That file is the source of truth; this page may be behind it.