A content-addressed file store on top of any S3-compatible bucket, with lazy chunked reads and a shared local disk cache.
Whole files live in the bucket under one name, which is the entire object key (optionally behind a configured prefix). Uploads are therefore idempotent, and two writers racing on the same content write identical bytes, so there is nothing to coordinate.
The name is sha256 of the file's own per-chunk hash list, so an artifact is self-authenticating at chunk granularity: every 4MB range the store serves is checked against that list before anyone sees it. See The name below.
Locally, an object is cached as plain 4MB chunk files under
<window>/<owner>/, where the window is a 20-minute UTC time bucket. Reads
are open+pread; a consumer that genuinely needs a range for the life of its
process can map it, one mapping per chunk.
Zero dependencies outside the standard library, apart from AWS credential
resolution. Linux only: the cache uses statfs and mmap with no fallback.
A stored object is three runs of bytes:
[ content: L ][ hash list: 32*N ][ trailer: 16 ]
N = ceil(L / chunk size) and chunk i is content[cs*i : min(cs*(i+1), L)].
The last chunk is short, never padded. list[i] is sha256 of chunk i. The
trailer is u64 LE of L followed by the 8-byte magic CASFSv1\n, magic last,
so tail -c8 identifies the format.
The name is sha256(hash list). Reading it works out in one round trip
because L + 32*ceil(L/cs) + 16 is strictly increasing in L, so a single
stored size admits exactly one L: the reader derives the tail's length from
the object size, fetches it, checks sha256(list) == name, and from then on
every chunk verifies positionally against list[i].
Two levels, no tree, no proofs. One list therefore describes at most 131072 chunks, 512 GB at the 4MB default; over that the format refuses rather than grows a level nothing would use.
The tail is metadata and no consumer ever sees it. File.Size() is L,
every range lives in [0, L), and the hash list and trailer are invisible above
this package. A footer-last file format therefore seeks from Size(), not from
the end of the file on disk.
The trade this buys: a bucket is genuinely untrusted infrastructure. It cannot substitute a chunk, cannot answer with a different object of the same size, and cannot serve a truncated or edited hash list. The only move it has left is to withhold bytes, which is what a mirror answers.
The store owns a spool directory. A caller adds content by atomically renaming
a sealed, correctly named file into that directory. That rename is the
registration. There is no handshake, no ack, no journal, no in-memory
bookkeeping that a kill -9 can lose. Either the rename happened or it did
not, and the filesystem already answered that question.
Sync scans the spool and uploads everything the bucket does not already have.
It can run at startup, on a ticker, or right after a write; a file dropped in
the spool by a process that then died is picked up by the next one to look.
Each upload makes ONE sequential pass over the file first, computing both the
object digest SigV4 signs and the hash list the name commits to, so a file
whose name lies about its contents is refused before a byte goes out. That is
the same check on the single-PUT road and the multipart one.
Release is the only thing that removes a spool file, and it refuses until the
bucket is confirmed to hold the content. So a hash is never unreadable: the
durable copy is the spool file, then the bucket, and the two overlap.
The cache never counts as durability. It is disposable by construction, which is what lets everything below be as blunt as it is.
Open(hash).ReadAt always reads through the chunk cache. Never anything else.
A chunk miss is filled from one of two sources, and that is the only difference between local and remote content:
- the spool file is still there, so
preadthe aligned range out of it - it is not, so fetch the chunk as
ceil(chunk / 1MB)parallel sub-ranges that tile it exactly, and assemble
The split is transport only: S3 answers in ~20ms whatever the range's size and
then moves ~100 MB/s per connection, so 4x1MB in parallel is ~31ms against
~63ms for one serial 4MB GET. Every sub-range is checked against the
Content-Range it comes back with, and one that fails, arrives short or
arrives from the wrong offset fails the whole chunk. Sub-range requests are
bounded store-wide by FetchConcurrency (default 16), not per chunk, because a
joining node misses on many chunks at once.
Concurrent misses on one chunk collapse into a single upstream read. Every
fill is verified against list[i] before it is cached or served, and a
mismatch is a loud error naming the artifact and the chunk index, never data. A
cache hit is not re-verified: the file was verified when it was filled and
landed by fsync + rename, and re-hashing 4MB on every hot read buys nothing.
Verified bytes are handed back to every waiter and, if the disk has room,
written to the current window as <hash>.<index> by tmp + fsync + rename.
The upload transition is invisible. Ranges that real traffic touched while
the file was spool-resident keep being served from the spool descriptor the
File holds, so when the upload ACKs and Release unlinks the spool file, the
hot path does not notice. Release stays dumb on purpose: it does not pre-warm
the cache. Ranges nobody ever read while the file was local are cold by
definition, and copying gigabytes of them in would only push out chunks
somebody else is using.
s, err := casfs.New(casfs.Config{
Endpoint: "https://<account>.r2.cloudflarestorage.com", // or http://127.0.0.1:9000
Region: "auto",
Bucket: "epochs",
Prefix: "v1/", // optional, used verbatim
AccessKey: "...", // static keys win; leave empty for the AWS default chain
SecretKey: "...",
SpoolDir: "/var/lib/app/spool",
CacheDir: "/var/lib/app/cache",
Namespace: "2oYMBNV4...FByM", // this store's owner inside the cache tree
ChunkSize: 4 << 20, // default
// Admission watermark, in absolute bytes of free space. 0 means 5% of the
// filesystem, i.e. cache up to 95% full.
CacheMinFree: 20 << 30,
// Whole windows older than this are deleted by name, whatever the disk
// looks like. 0 means 30 days.
CacheMaxAge: 30 * 24 * time.Hour,
})
// Streaming producer: hash content as you write it, append the tail, rename.
h := casfs.NewHasher(4 << 20)
io.Copy(io.MultiWriter(file, h), content)
name, tail, err := h.Finish() // name == sha256(hash list); tail = list+trailer
file.Write(tail) // ...then fsync and rename onto SpoolPath(name)
path := s.SpoolPath(name) // rename your sealed file onto this, yourself
name, err := s.Put(path) // or let Put seal it and do the rename for you
done, err := s.Sync() // upload everything spooled and not yet in the bucket
err = s.Release(hash) // unlink the spool file, refuses until uploaded
f, err := s.Open(hash) // *File: io.ReaderAt, io.Closer, Size() int64 = L
n, err := f.ReadAt(p, off) // the query path: pread and copy
v, err := f.View(off, n) // long-lived: one mapping per chunk
b := v.Slice(off, n) // aliases inside a chunk, copies across one
err = v.Close() // unmaps
err = s.Close() // stops the eviction worker, nothing else
err = s.SetPointer("latest", hash)
val, err := s.GetPointer("latest") // fs.ErrNotExist if absent
st := s.Stats() // evictions, admission refusals, victim age, free bytesPut is a convenience: it streams the file to build its hash list, appends the
tail, and renames it into the spool, and requires the file to be on the same
filesystem as SpoolDir. A caller that streams its own artifact should use
NewHasher while it writes, append the tail Finish returns, and do the rename
itself; nothing on either road holds the artifact in memory. Both end at the
same rename.
Sync returns the hashes now confirmed present in the bucket, so the usual
loop is for _, h := range done { s.Release(h) }. Errors are collected per
file, so one bad spool entry does not stall the rest.
Pointers are the one mutable, non-content-addressed object. A pointer name that
looks like a content hash is rejected, so the two key spaces cannot collide.
They have EXACTLY THE SPOOL SEMANTICS OF CONTENT. SetPointer writes the value
under the spool (.pointers/<name>, tmp+rename) and returns, making no network
call at all; Sync uploads it AFTER that pass's content, so a bucket reader
following a pointer never lands on an object that is not there yet, and then
deletes the local file, the same release an artifact gets. A pass whose content
upload failed leaves its pointers spooled and retries next time.
GetPointer answers from the spooled value while there is one. Once it has
been uploaded and released, or on a store that never wrote it, the read goes to
the bucket and the value is written back to the spool, so the next read and the
next process are local again. That write-back is clean, never dirty: a store
re-uploading a value it did not author could only put a stale pointer over a
newer one. A bucket read that fails bubbles as it is, and a missing pointer
wraps fs.ErrNotExist; nothing is ever invented.
A store whose credentials have expired therefore keeps setting and reading its own pointers exactly as an offline one does. Only uploads stall, and the values simply stay in the spool until they can go.
Close stops the eviction worker and does nothing else. There is no flush, no
marker, and no shutdown handshake: skipping it costs nothing at all.
<SpoolDir>/<hash> durable until uploaded
<SpoolDir>/.pointers/<name> durable until uploaded
<CacheDir>/<window>/<owner>/<hash>.<index> disposable, 4MB per file
<CacheDir>/<window>/<owner>/<hash>.<index>.*.tmp a fill in flight
A chunk file is named by the artifact hash and the chunk index, not by a content hash of the chunk. That name already identifies immutable bytes uniquely, a reader can derive it without anyone's chunk list, and any process looking at the file can decide who it belongs to. A content-hashed name would buy deduplication nobody has ever needed here and cost the ability to find a chunk you have not already been told the hash of.
The window comes first, and that ordering is the whole design. A window is
the UTC start of a 20-minute bucket, written 2026-08-03T12-40: fixed width in
every field, so alphabetical order is chronological order. The globally
oldest cohort is therefore the lexicographically first directory under the
cache root, which is what collapses eviction into rm -r of one directory. It
also means an operator can ls the tree, read the cache horizon off the first
entry, and hand-trim it.
The owner (Config.Namespace) is the level under it. Hash filenames are
illegible to a human, so this exists to make ownership visible: du per owner,
rm -r <CacheDir>/*/<owner> to wipe exactly one owner's cache, and the
one-writer-per-owner invariant enforced by the directory tree instead of by a
convention. Several owners share one cache root and one set of windows, so they
also share the eviction horizon, which is the point when they are chains in one
process competing for one disk.
The window a file sits in is its recency, so the LRU lives in the directory tree rather than in anyone's memory: it survives a restart with no journal, and several processes can share it without agreeing about anything.
New does one name-only walk of its own owner directory in each window to
build this process's chunk -> window map (175k files in ~112ms warm on ext4).
Anything at the cache root that is not a window directory is deleted, which is
the entire migration story off any older layout. There is no compatibility
mode: the cache is disposable, so a layout change is paid for by refetching.
That map is a hint owned by one process. An entry that is wrong because a
sibling process evicted the file costs one ENOENT and a refetch, never a
wrong answer.
Reading a chunk that is in any window older than the current one renames it
into <current window>/<owner>/, creating the directory on demand. That is at most one rename per chunk per window, and it
is deliberately greedy: promoting only when a chunk is two or more windows
behind stops distinguishing hot from cold exactly when it matters, because
under pressure everything the worker is about to eat looks equally stale, and
the cache degrades to FIFO.
The file is opened before the rename, so a promotion that loses a race to another process costs a map entry and never the read in flight.
Before a fill is written, statfs says how many bytes are available. Over the
watermark, the chunk is not cached: the fetched bytes are served from
memory and forgotten. Nothing is ever deleted to make room for a fill.
The fill itself is tmp + fsync + rename. The fsync is not tuning. Power loss between the write and the rename must never leave a torn chunk under a correct name: a cache hit is not re-hashed, so torn bytes under the right name are a silent wrong answer, and a half-written bloom page answers "definitely absent" for keys that are there. A tmp name never becomes a chunk name, so a torn write is only ever garbage to collect.
The watermark is absolute bytes, not a percentage, because casfs does not own the filesystem: what matters to everything else on the box is how much room is left, not what share of it casfs took. The default is 5% of the filesystem free, i.e. cache up to 95% full.
One goroutine per store. While the disk is over the watermark it drops the
oldest window entirely and then waits a minute for statfs to settle before
believing it again; otherwise it idles on a short poll. There is no victim
selection, no cursor, and no scan: the names sort chronologically, so the first
entry of a sorted readdir is the oldest cohort, and everything in it is at
least one window cold by construction. Every owner sharing that window goes
with it.
It never touches the current window. If the current window is all there is, the worker stops and admission control carries the load: a saturated cache freezes full rather than eating the fills it just made. Churning is strictly worse than freezing, because every evicted-and-refetched chunk is a GET that bought nothing.
The accepted edge, recorded rather than hidden: a cache filled fast (a bootstrap, a backfill) concentrates most of its bytes in a few windows, so the first pressure eviction after one drops a big cohort at once. It is deliberate and self-correcting, it only fires when the disk is genuinely full, and everything it drops was already at least a window cold.
Separately, and regardless of disk fill, whole windows older than
CacheMaxAge (default 30 days) are dropped by name, with no stat and no
per-file work: the same operation, gated on the window's own name instead of on
the watermark. That is safe precisely because a promotion's target is always the
current window, so nothing anyone still reads can be sitting in a directory
named thirty days ago. Stray *.tmp files older than an hour are collected in
the same pass.
Stats() reports admission refusals, the number of windows dropped, and
the age of the window the worker last dropped. That last number is the
honest cache horizon: how long a chunk actually survives here. A byte budget
could only ever have reported its own configuration back.
It also reports the failures that have no caller to return them to, because
every one of them still produces a right answer and is therefore invisible:
FillErrors (a fill that could not land), EvictErrors (the worker could not
measure or free space) and CacheReadErrors (a chunk that WAS cached but
would not read or map, so the read went to the network instead). A node
climbing on any of these is an S3 passthrough with a healthy-looking hit
story, and LastError names the most recent one.
ReadAt is the query path and it copies. Use it for everything transient.
View(off, n) is for ranges a consumer reads for the life of its process:
indexes, dictionaries, filters. It maps one chunk file per chunk the range
covers. Slice is zero-copy inside a chunk and copies across a chunk
boundary, which is the whole price of chunk files not being contiguous: at
4MB granularity a 69-byte index entry straddles one boundary in about 60000, so
the copy is noise and the common case allocates nothing.
Because a mapping keeps its inode alive, a chunk that is unlinked while mapped
keeps its blocks until the last Close. The ghost disk a consumer can hold is
therefore bounded by the size of its resident set, and that is the accepted
cost of not having a pin protocol. In exchange there is no way for eviction to
turn live bytes into zeros, which is what the old hole-punch cache had to be
defended against with pins.
A View of a spool-resident object maps the spool file directly, so it never
doubles the bytes on disk. A chunk the disk refused to cache is held as heap
bytes instead, so a view is always complete whatever the disk is doing.
There is no AdoptDir, because the spool directory already is one. Point your
writer at SpoolDir, name files after their hash, call Sync on a ticker, and
files trickle to S3 at whatever pace you tick. That is the whole feature and it
needs no library code.
Hand-rolled SigV4 over net/http: casfs needs exactly HEAD, PUT, GET and
ranged GET, which is about 60 lines of signing that both R2 and MinIO accept,
versus minio-go's transitive dependency tree for the same four calls.
Over 5 GiB a single PUT is refused (EntityTooLarge), so an artifact past that
goes up as a multipart upload: initiate, one PUT per 128 MiB part, complete.
Each part is buffered whole because SigV4 commits to its sha256 before the body
goes out, so peak memory is one part regardless of the object, and parts are
sequential because the bottleneck is the uplink. Two S3 behaviours the code
takes seriously: CompleteMultipartUpload can fail with an Error body under a
200 OK (the status line is written before the assembly happens), and anything
that fails after initiate leaves billable parts behind, so it aborts on the way
out and returns the original error. Multipart and single PUT send the same
bytes and produce the same name, because the name is a function of the content
and never of how it went up.
The one AWS dependency is credential resolution, not S3: aws-sdk-go-v2's
config/credentials supply Config.Credentials, so SSO, instance roles and
anything else the default chain knows about work, including their refresh.
Static AccessKey/SecretKey take precedence and never touch the chain (the
R2 and MinIO path). Credentials are retrieved per request; when they carry a
session token it is signed as a fourth header, x-amz-security-token, and
TestSignMatchesAWSSigner compares the whole Authorization header against the
SDK's own signer with and without one. The default chain also supplies the
region when Region is empty, before the "auto" fallback.
- Artifact identity is not persisted.
Openon a hash that is neither spooled nor known to this process issues one HEAD plus one ranged read of the tail, then keeps the hash list resident (32 bytes per 4MB, so 76KB for a 9.5 GB artifact). That read also carries the last content chunk back, which is a chunk the caller was about to ask for anyway in any footer-last format. - The query path opens the chunk file on every
ReadAt. That isopen,pread,close: three syscalls, no descriptor table to bound, and no cache to invalidate when a sibling process deletes the file underneath. - Over the watermark every read refetches, because nothing is allowed to land.
That is a real degraded mode and it is counted (
Stats().Refusals) rather than hidden. - Not implemented: multipart upload, DELETE, LIST, retries and backoff, and virtual-host-style addressing.
go test -race ./... runs entirely in process against a fake S3 built on
httptest. No docker, no network. The fake verifies that requests carry a
well-formed SigV4 Authorization header and that PUT bodies match the signed
payload hash; the signing key derivation is pinned to the published AWS test
vector. Signature acceptance by a real service is not covered offline.
The cache is tested against the filesystem rather than against itself. The
worker is asserted to drain the oldest window first, whole, and then to stop
dead when only the current window is left. Admission refusal is driven by a
fake statfs, and a view over a disk that refuses everything is asserted to
still return the right bytes. A chunk planted a day back in the window tree is
asserted to be renamed forward by the read that touches it. A truncated tmp is
asserted never to become a named chunk, and the read that follows it is
asserted to come back correct. Thirty-two goroutines racing on one cold chunk
are asserted to cost exactly one GET, and eight readers running against a
goroutine dropping whole windows in a loop are asserted to return byte-correct
answers throughout.
The format has its own tests, and they are the adversarial half. A chunk whose bytes are substituted in place, same length and same offset under the same name, is asserted to be REJECTED rather than served, and never cached. A different artifact of exactly the same size served under this name is refused at open. A hash list with one flipped bit, a truncated tail and a missing trailer all fail. An artifact over the 131072-chunk ceiling is refused while exactly at the ceiling is accepted, and the short last chunk round-trips at every length around a boundary.
An upload that FAILED is never called corruption: the file is checked in a pass of its own before the PUT, so a PUT that died early (expired token, dropped connection) reports the upload failure and nothing else. An operator never hears "corrupt" about content that is fine.