Narad v1.2.0

Idle log eviction

Topics that were used once and abandoned used to hold their runtime cost forever: every opened partition log kept two goroutines (a 100ms-tick flusher and a retention reaper), open segment file descriptors, and buffer/index memory until the topic was deleted or the process restarted.

Narad now closes any partition log untouched for storage.idle_log_eviction_ms (default 30 minutes, 0 disables, floor 60s — file-configurable) and reopens it lazily, invisibly, on the next produce, consume, or replay (#103).

What makes it safe:

• Observation never keeps a log warm. Metrics polls peek without opening, and a fan-out child that is attached but silent no longer pins its parent: caught-up cursors are answered from the durable high-watermark file (force-synced on every clean close) without reopening the log.
• Committed backlog always opens the log — correctness over eviction, always. A record produced while a cursor waits against a closed log is picked up within 250ms.
• Retention is never stranded: eviction defers until the reaper has finished deleting aged segments.
• No races with produce: the close runs under the same locking discipline as topic deletion, re-verified under lock; a produce-vs-evict hammer test (eviction forced on every sweep) loses zero records under the race detector.

New metrics: narad_open_partition_logs, narad_idle_logs_evicted_total.

Also: the config docs no longer show locked storage internals in the example file — the strict loader rejects them, and the example now matches (data_dir, codec, compression_level, idle_log_eviction_ms are the four accepted keys).

Create short-lived topics and forget to delete them? Now rude but free.

🤖 Generated with Claude Code

Narad v1.1.0 — replication, when you ask for it

One API call:

curl -u $AUTH -X POST $NARAD/v1/topics \
  -d '{"name": "orders-replica", "parent": "orders"}'

That creates a fan-out child that is an asynchronous full copy of the parent — and, new in this release, its partitions are deliberately placed on different nodes than the parent's. A keyed record's original and copy never share a disk. Lose the parent's volume, and the data lives on in the replica (RPO ≈ fan-out lag, typically sub-second).

No replication subsystem was built for this — no quorums, no consistency protocol. It's a placement rule on the fan-out machinery that already soak-tested for days at 1,000 msg/s. You pay double disk and write IO only on topics that opt in, and a replica with longer retention doubles as an archive tier.

What's in it

• Anti-affine placement (#101): a fan-out child's partition p is assigned away from the owner of the parent's partition p, in both assignment paths (create/alter and the controller reconcile sweep), sharing one decision function so they can never diverge. Deferral instead of guessing when the parent isn't placed yet. Existing assignments never move.
• Create-as-child: POST /v1/topics accepts parent (+ fanout_delay_ms) — create → attach → assign in one call, which is the only moment anti-affinity can act (assignments are sticky). Partitions default to the parent's count; a failed attach rolls the create back; requires manage rights on the parent. CLI: --parent / --fanout-delay-ms. Works through the leader-forward path (#102 — caught by the post-deploy live check, plus a handler↔RPC field-parity test so the class of drift stays dead).
• owner_node in partition stats: see the guarantee yourself by comparing parent and replica stats.

Honesty section

It's DR, not HA: the copy is async (records not yet fanned out die with the parent's volume) and consumers switch to the replica themselves — nothing fails over automatically. Children attached the pre-existing two-step way keep their old placement; use one-call creation for the pattern. Guaranteed separation needs ≥2 live nodes and matching partition counts (the default).

Upgrade notes: fully backward compatible — no wire or on-disk format changes; existing assignments untouched. Docs: https://debanganthakuria.github.io/narad/client/fanout-and-delay/#replication-when-you-ask-for-it

🤖 Generated with Claude Code

Narad v1.0.0

Narad graduates to v1.0.0 — first production release.

What Narad is

A WAL-first, AP message broker in a single Go binary. Hit any node to produce, consume, or ack; the cluster does the routing. Queue semantics with visibility leases, fan-out topics, delayed delivery, offset replay, per-topic retention, zstd on disk, Raft-backed metadata. No ZooKeeper, no page of tuning knobs.

Send anything. Produce is a raw octet-stream: JSON, plain text, protobuf, binary — no client-side encoding, no SDK. JSON comes back verbatim, text as text, binary base64-flagged with payload_encoding (#100).

The evidence behind the 1.0

Everything below ran against a live 5-node cluster on Kubernetes.

Soak windows at 1,000 msg/s, full produce→consume→ack flow across parent + fan-out child + 60s-delay child:

• Final window: 47h 28m, 170.9M messages, zero lost, zero produce failures, zero consume errors (4 produce retries total). Delay path fired 0 messages early, 0 overdue.
• Prior windows: ~37h (~133M messages, ended by infrastructure node churn — broker self-healed, zero loss) and ~23h validation windows. 300M+ messages soaked in aggregate.

Chaos matrix — kill -9 of owners, leaders, joiners; restarts mid-produce; all zero-loss.

Crash-recovery hardening — the stale-replica defense family (#71–#74): leader confirmation before destructive reconciliation, freshness-gated catch-up checks, and fresh leaders must Barrier + re-read before trusting their FSM.

Capacity — 50,000 msg/s sustained through the full produce→consume→ack flow on a 3-node cluster; the ceiling was the load generator, not the broker.

Operational drills — mixed-version rolling upgrade (partitioned StatefulSet roll), full backup/restore with demonstrated RPO, cluster scale-out 3→5 live under load, and an offset-replay drill (500 exact-offset reads + determinism checks + 2,000-read hammer) with no impact on live consumers.

Notable since the last beta

• Payloads of any content type consumable end-to-end: JSON verbatim, text as string, binary base64-flagged (#100).
• Ack backpressure: when a partition's acked-ahead set is full, ReserveNext serves only the frontier hole instead of feeding a redelivery spiral (#91).
• Replay error contract: offsets reaped by retention return 410 Gone, negative offsets 400 — not 500s (#99).
• Cluster scale-out via Raft join for fresh nodes (#75, #76).

Docs

Client guide, operator handbook (Helm chart included), and code-level internals: https://debanganthakuria.github.io/narad/

Honesty section

Narad is AP: it stays available and durable through node loss, and in exchange does not guarantee ordering. Partitions are single-owner with no replication — a permanently lost node loses its unconsumed partitions (retention-window data). Rebalance-on-join and node decommission are not in 1.0; new nodes serve partitions of topics created after they join.

🤖 Generated with Claude Code

v0.2.0-beta.4 — ack-503 spiral breaker

One fix, found by soak analysis rather than a bug report: a consumer that doesn't retry failed acks could, under backlog, trap a partition in a redelivery spiral — acked-ahead set full → every ack 503s → deliveries expire → redeliver → repeat (observed live: 623k duplicate deliveries over 7 hours; zero loss, but nobody's idea of steady state).

ReserveNext now refuses to hand out fresh offsets while a partition's acked-ahead set is at capacity: the only reservable offset is the frontier hole itself, whose ack collapses the run and restores normal service. The Consuming docs now also state the client contract plainly: retry your 503 acks.

Drop-in upgrade from beta.3; no API or on-disk changes.

v0.2.0-beta.3 — chaos-hardened durability + cluster scale-out

Everything in this release came out of chaos testing the previous beta on a live cluster under continuous soak traffic: force-kills of followers, leaders, two nodes at once, and kills mid-rolling-restart. Two more members of the stale-replica bug family were found and fixed, the last known disk-hygiene follow-up shipped, and the cluster can now scale out.

Fixed

Dispatcher could discard accepted records on a stale replica view (#73)

The produce dispatcher discarded WAL records — 202-acked, durable, undelivered — when their topic was absent from the local metastore replica. A replica restored from an old Raft snapshot is missing every topic created after the snapshot point, so a restarted node could permanently destroy accepted records for live topics. Discard now requires local absence and a caught-up replica and the leader confirming the topic is gone; every failure mode keeps the records.

A freshly elected leader could trust its still-replaying FSM (#74)

The subtlest one, caught by a simultaneous two-pod kill: election guarantees a new leader's log is complete — not that its FSM has applied it. The node that won the election after a force-kill confirmed a dead fan-out attach epoch from its own snapshot-restored FSM, tail-anchored its cursors, and silently skipped ~2,100 due deliveries; the follower that restarted alongside it asked the real leader, was refused, and lost nothing. Every "I am the leader, my state is authority" shortcut now runs a Raft Barrier and re-reads local state after it. The fan-out cursor-file sweep also now requires leader confirmation per removal, and consumer offsets recover lazily from the per-partition file (disk is ground truth) instead of a boot-time scan over possibly-stale assignments.

Added

Ingress WAL auto-reclaim (#73)

A fully-dispatched active WAL segment was pinned on disk until new appends overflowed it — never, once its topics went quiet (observed: 5–51 MB of dead WAL per node). Compaction now rotates a fully-dispatched active segment past a 1 MiB floor and reclaims it at the normal checkpoint cadence; recovery preserves the sequence space through an empty rotated log. No manual cleanup, no startup pass.

Cluster scale-out (#75, #76)

Raft membership was bootstrap-only; scaling the StatefulSet produced phantom clusters. Now: initialClusterSize (set once) gates bootstrap, any other fresh node starts join-only — empty Raft configuration, walks its peers with the new OpJoinCluster RPC until the leader admits it via AddVoter, and holds readiness until admitted so an unjoined node never receives traffic. Scaling out is --set replicaCount=N. Existing partitions are never rebalanced (single-owner design); new nodes take assignments for topics created after they join. Verified live 3→5: admission in seconds, perfect round-robin spread of a new topic, and a leader+new-node double-kill at 5 nodes held quorum with zero message loss.

Verification

Chaos matrix on a live cluster at 100 msg/s × 3 paths (direct, fan-out child, 60s delay child) with a Redis-ledger harness detecting any lost message: leader kill, child-partition-owner kill, two-node quorum loss (three runs), kill during rolling restart, 3→5 scale-out, and a five-node double-kill — zero lost messages end-state in every scenario, bounded at-least-once duplicates only.

Upgrade notes: no API or on-disk format changes; drop-in from v0.2.0-beta.2. To enable scale-out on an existing install, upgrade the chart (adds NARAD_CLUSTER_INITIAL_MEMBERS; default initialClusterSize: 3 matches existing clusters), then raise replicaCount. Scale-down/decommission is not yet supported.

v0.2.0-beta.2 — crash-recovery hardening

Two data-loss bugs in crash recovery, found by force-killing pods under live soak traffic and verified fixed the same way. Both share one root cause: a replica restored from a Raft snapshot reads as "caught up" against its own local indexes while its FSM is hours stale. Both fixes share one philosophy: destructive actions require leader confirmation, and every failure mode keeps the data.

Fixed

Startup orphan sweep could delete live topic data (#71)

A freshly restarted node's sweep compared on-disk topic directories against its stale topic set and reclaimed live topics' partition data seconds after boot. A directory is now deleted only when the leader confirms the topic is absent (new OpGetTopic peer RPC). No leader, unreachable leader, non-404 answer — the directory stays.

Fan-out cursors could silently rewind to tail, dropping the delay backlog (#72)

The fan-out runner reconciling against a stale FSM spawned cursors under dead attach epochs, tail-anchored on the epoch mismatch, and overwrote the good offset files — so the post-catch-up cursors re-anchored at the current tail and silently skipped everything pending (measured live: ~2,000 lost delay-child deliveries at 100 msg/s with a 60s delay). Three layers now prevent it:

• AppliedCaughtUp additionally requires fresh leader contact (raft LastContact ≤ 5s) on followers.
• The runner skips reconcile passes — spawn and cleanup — until the replica is caught up, and a stopped cursor may only remove an offset file whose epoch it owns.
• Tail-anchoring (destructive by definition: skips records, overwrites the offset file) requires the leader to confirm the attach epoch; any failure defers the cursor and retries at reconcile cadence. Clean resume from a matching file needs no RPC.

Verification

Force-kill under 300 msg/s soak traffic (parent + fan-out child + 60s delay child, Redis-ledger loss detection): topic dirs kept, cursors resumed from their offset files through a mid-catch-up stale window, delay backlog delivered in full — zero lost messages, OVERDUE=0, only bounded at-least-once duplicates from the in-flight slab.

Upgrade notes: no API, envelope, or on-disk format changes; drop-in upgrade from v0.2.0-beta.1.

Narad v0.2.0-beta.1 — feature complete

Narad's feature-complete milestone: the beta line begins here. Everything below is running on the dev cluster and verified end to end.

Topic fan-out (parent → child) — #68

A parent topic replicates every message into up to 108 independent child topics. Producing to a parent stays byte-for-byte a normal produce; per-(child, partition) cursors on the parent partition owners tail the committed log, re-key with each child's partitioner (per-key ordering preserved within a child), and batch-commit with a commit-before-advance at-least-once contract. Attach epochs guarantee detach/re-attach never replays. Schemas are inherited and parent-managed while attached.

Delay children — #70

Attach a child with delay_ms and every record is delivered only once parentCommitTime + delay has passed — retry-backoff tiers and scheduled reprocessing from one flag. Commit times are monotonic per partition, so the due check is a head-peek-and-sleep: O(1) while idle, pending backlog costs disk, not memory. The parent must retain delay + 1h (enforced at attach and on retention shrink); direct produce to a delayed child is rejected; narad_fanout_due_lag_seconds is the health signal.

Ack lease operations — #69

POST /ack?extend=true renews the visibility window (slow-consumer heartbeat, same receipt handle); extend=0 is a NACK — immediate redelivery under a fresh handle, waking parked long-pollers. Both share ack's validation: a lapsed lease returns 410 and can never be revived. Hot-path benchmarks unchanged.

Also in this release

• Uniform 1-hour retention floor for every topic (the fan-out/delay buffer guarantee).
• Keyed record envelope (v2): stored records carry produce key + commit timestamp; consumers now receive Message.Key and real commit-time Message.Timestamp.
• New metrics: fan-out lag/committed/dropped/due-lag, batch histograms, narad_ack_extended_total, narad_nack_total.
• CLI: topics attach [--delay-ms] / detach / children, ack --extend / --nack.

Breaking / upgrade notes

• Storage format: the partition-log record envelope changed (v2, versioned). Pre-beta partition data does not decode — wipe topic data when upgrading from any alpha build. Ingress WAL and metastore are unaffected.
• Attach/detach cluster RPCs changed shape; run homogeneous versions (no rolling mixed-version clusters across this boundary).

Verification

1424 tests green (unit, e2e, 3-node integration, chaos), race-detector clean, three adversarial review passes with all findings fixed, and full live verification on the dev cluster: 31 fan-out checks, 35 lease checks, 30 delay checks including a live 20s due-gate timing proof.

Narad v0.1.0-alpha.2 — the security release

Second alpha of the v0.1.0 line. The headline is security: Narad now has full authentication, RBAC, and encrypted+authenticated intra-cluster transports — and it's secure by default. Still pre-1.0 and not production-ready (see the honest gate list below); suitable for evaluation and early integration behind a trusted boundary.

Image: ghcr.io/debanganthakuria/narad:v0.1.0-alpha.2 (linux/amd64 + linux/arm64).

🔐 Security (new)

• HTTP API authentication — HTTP Basic auth over a bcrypt-hashed user store, with a per-node verification cache (positive/negative caches + decaying failed-attempt throttle + singleflight) so the hot path never pays bcrypt's cost. Secure by default; a root admin is seeded at first boot (random password logged once, or NARAD_ADMIN_PASSWORD).
• RBAC — users with produce / consume (incl. ack) / create / admin grants over literal or prefix-wildcard (orders-*) topic patterns. Full user-management API (/v1/users) with no-privilege-escalation invariants (only root confers admin; nobody edits their own grants; root is undeletable/immutable).
• Topic ownership — the creator owns a topic; only the owner or an admin may alter/delete it (distinct from produce/consume, which are grant-governed).
• Cluster RPC auth — the node-to-node QUIC transport (already TLS 1.3) now requires a shared-secret HMAC handshake per stream.
• Raft metadata transport mTLS — mutual TLS on the consensus channel that replicates user hashes and grants; peers authenticate by cluster-CA-signed certs. Optional, off until certs are provisioned.
• Pentested on a live cluster: 44/44 designed invariants held; the one finding (public /metrics exposure via the ingress) is fixed.

🛠 Reliability & correctness

• 40-bug correctness pass across storage, WAL, cluster, and broker.
• Multi-arch images fixed — arm64 images now contain arm64 binaries (previously every arm64 image shipped an amd64 binary and crash-looped).

🧹 Internals

• Deep readability refactor (file/function splits, dedup) with no behavior change.
• Dropped the inert SyncBytes knob and pruned debug-only metrics to the operator-useful set.

⚠️ Upgrade / breaking notes

• Secure by default. Existing unauthenticated clients will get 401 after upgrade. Provide credentials, or set NARAD_SECURITY_ENABLED=false for local/dev.
• Config decoding is strict — a removed key (e.g. ingress_wal_sync_bytes) now errors; delete it.
• Enabling Raft mTLS on a running cluster briefly pauses consensus writes mid-rollout (plaintext and mTLS nodes can't talk); reads continue.

🚧 Not production-ready — remaining gates

This alpha closes the security gate. Still open before prod:

• Durability / DR (hard gate) — no tested backup/restore runbook or measured RPO/RTO for this single-copy (no-replication) design.
• Soak & SLOs — only a point-in-time ~50k msg/s benchmark; no multi-day soak under fault injection against written SLOs.
• Upgrade/rollback contract — on-disk and node-RPC formats are current-version-only; mixed-version rolling upgrades are undefined.
• Partition rebalance — ownership is sticky; new capacity doesn't absorb existing partitions.

Full changelog: v0.1.0-alpha.1...v0.1.0-alpha.2

Narad v0.1.0-alpha.1

Narad's first alpha release.

This is a pre-1.0 release intended for early evaluation, local development, and design review. The README documents the current production-readiness gates: API auth/rate limiting/TLS and a tested durability/DR contract are still required before direct production or externally exposed deployments.

Container image: ghcr.io/debanganthakuria/narad:v0.1.0-alpha.1