v0.3.0 — erasure coding and self-healing repair

The third Hamster release: erasure coding and self-healing repair — and the S3 endpoint joins the cluster.

Dev preview. Objects now live erasure-coded across a real cluster, but the S3 write path commits only on the Raft leader (clients retry elsewhere), multipart and server-side copy are not on the cluster path yet, the data-plane membership is effectively fixed once data exists (rebalance lands in v0.4), and on-disk/on-wire formats may still change between v0 releases. Please don't trust real data to it — but the cluster preview now stores objects, not just metadata.

What's in v0.3

• Erasure-coded objects, end to end. A PUT erasure-codes the body into k+m self-describing shards spread across k+m distinct nodes (no two shards of one object ever share a node); a GET reconstructs from any k. Storage profiles follow an auto ladder as the cluster grows, with a small-object k=1 rule. Object data never passes through the Raft log — only the small metadata commit does, which is the design's first invariant.
• The S3 endpoint joins the cluster. hamster cluster run -s3 <addr> puts the full S3 API on every node: reads from the local replica, mutations as Raft proposals, objects through the erasure-coded data path. (Leader-only writes for now — a non-leader answers 503 SlowDown and clients retry elsewhere; multipart and server-side copy are refused on this path until their erasure-coded design lands.)
• The write-ack rule, mechanically. All k+m shards durable on the healthy path; a degraded write acks at a hard floor of k+1 and refuses below it with SlowDown. The metadata commit is the linearization point and happens only after the ack rule is met — so an object's durability budget is honest at the moment it is acknowledged.
• Self-healing repair. A repair sweep scrubs every shard against its replicated checksum and rebuilds missing or bit-rotted shards from any k verified survivors. Corruption is found by hashing and never laundered into a rebuild; a rotted shard is healed without anyone having to read the object first.
• Bounded memory, random-access reads. Windowed shard transfer keeps memory bounded per in-flight transfer regardless of object size, and a ranged GET moves only the shard slices it actually covers.

How it's verified

• The deterministic simulation harness drives the whole data path — placement, shard transfer, the ack rule, repair — through seeded cluster schedules: crashed receivers, down nodes, floor refusals, mid-PUT coordinator loss, degraded reads through crashed holders, an emptied node healed, two-shard bitrot rebuilt without a read, beyond-tolerance loss reported honestly, and crash-mid-sweep convergence. Durability is checked by decoding shard files off the surviving disks, not by trusting an ack.
• A six-node end-to-end suite runs real processes over loopback mTLS, stores objects 4+2, and kills nodes mid-workload: reads reconstruct around the loss, writes ack at the floor, and once the cluster is below the floor, writes refuse honestly while reads keep serving at exactly k.
• The race detector and the v0.1 compatibility suite (aws CLI, rclone, restic, s3cmd) keep passing.

Binaries below are static (CGO_ENABLED=0), version-stamped (hamster version), with SHA-256 checksums in SHA256SUMS. Next up, v0.4: partitioned placement made real — a stored, versioned cluster layout with zone-aware spread, capacity weighting, and rebalance, so a cluster can finally grow its data-plane membership after data exists.

v0.2.0 — clustering foundations

The second Hamster release: clustering foundations.

Dev preview. The metadata cluster is real and runnable; the S3 endpoint stays single-node until erasure coding gives objects a replicated home (v0.3). On-disk and on-wire formats may change between v0 releases. Try the cluster preview — three terminals, about two minutes.

What's in v0.2

• Raft-replicated metadata: one consensus group, every committed write on every member, rebuilt bit-identically from snapshot plus log on restart. Object data will never pass through this log — that's the design's first invariant.
• hamster cluster: init founds a cluster, token mints single-use, TTL-bound join tokens, run -token joins and runs in one restart-safe command, status shows membership and leadership from any node, recover is the explicit, documented exit from permanent quorum loss.
• Mutual TLS between nodes, always: init mints a cluster CA, joiners authenticate the cluster through the token's pinned CA hash before trusting anything, and every inter-node byte rides mTLS with certificate-bound identity. There is no plaintext mode and no flag to create one — and nothing to configure.
• Learners and the voter cap: nodes join as learners and are promoted to voters automatically once caught up, capped at five voters so quorum cost stays constant no matter how large the cluster grows.
• Crash-safe compaction: snapshot-carrying single-frame log rotation; a lagging or brand-new member catches up via a streamed snapshot that carries the address book with the data.

How it's verified

• The deterministic simulation harness runs the consensus layer through seeded schedules: elections, leader crashes, partitions, snapshot catch-up, membership growth, voter-cap enforcement, recovery from quorum loss — including the case where recovery must preserve a write the dead majority may have committed. Election timing is Hamster's own and fully schedulable, so the same seed replays the same history, elections included.
• An e2e suite drives the real binary through the operator's lifecycle: init, token joins, status, a leader killed mid-flight, failover, restart from disk, clean shutdown.
• The race detector and the v0.1 compatibility suite (aws CLI, rclone, restic, s3cmd) keep passing.

Binaries below are static (CGO_ENABLED=0), version-stamped (hamster version), with SHA-256 checksums in SHA256SUMS. Next up, v0.3: erasure coding with self-healing repair — and the S3 endpoint joins the cluster.

v0.1.0 — the single-node store

The first Hamster release: the single-node store.

Dev preview. Single node, no erasure coding yet, and on-disk formats may change between v0 releases. Please don't trust real data to it — but please do kick the tires: the quick start takes about a minute.

What's in v0.1

• Core S3 API: objects (PUT/GET/HEAD/DELETE with Range), both ListObjects versions, multipart uploads end to end, server-side copies (CopyObject, UploadPartCopy), batch deletes, presigned URLs. Anything outside the surface returns an honest NotImplemented instead of a wrong answer.
• Full SigV4 authentication: Authorization header, presigned query, UNSIGNED-PAYLOAD, and aws-chunked streaming signatures — implemented in-house on the standard library, conformance-tested against AWS's published examples.
• Both addressing styles: path-style and virtual-hosted (-domain flag).
• ETags exactly like S3: MD5, including the composite -N form for multipart — the semantics sync tools verify against.
• Streaming write path: uploads go to disk through a bounded write buffer; a 1 GiB PUT peaks the server around 12 MB of memory.
• Durable storage: BadgerDB metadata and synced blobs; every on-disk record is additively versioned protobuf with golden-pinned encodings, so today's data stays readable by every future version.

How it's verified

• A third-party client compatibility suite runs the real aws CLI, rclone, restic, and s3cmd against every change.
• A deterministic simulation harness boots the store on a simulated disk with real crash semantics, kills it adversarially across seeded schedules, and checks every recovery against a reference model.

Binaries below are static (CGO_ENABLED=0), version-stamped (hamster version), with SHA-256 checksums in SHA256SUMS. Next up, v0.2: clustering — Raft-replicated metadata and multi-node membership.