cfsread

Cached file reader with transparent decompression and singleflight coalescing for Go.

What it does

cfsread reads files from any io/fs.FS source, transparently decompresses them based on magic-byte detection, and caches results in a bounded LRU cache. Concurrent reads of the same file are coalesced via singleflight so only one goroutine performs I/O.

Designed for high-throughput, read-mostly workloads — static asset serving, configuration loading, template rendering — where the same files are read repeatedly.

Installation

go get github.com/lbe/cfsread

Quick start

package main

import (
    "context"
    "embed"
    "fmt"
    "io/fs"

    "github.com/lbe/cfsread"
    "github.com/lbe/cfsread/decompress"
    "github.com/lbe/cfsread/decompress/lz4"
)

//go:embed static/*
var embedded embed.FS

func main() {
    sub, err := fs.Sub(embedded, "static")
    if err != nil {
        panic(err)
    }
    src := cfsread.Source{ID: "app", FS: sub}

    // Register decompressors (optional — plain files pass through).
    reg := decompress.NewRegistry()
    if err := reg.Register(lz4.New()); err != nil {
        panic(err)
    }

    r := cfsread.New(cfsread.Options{
        MaxEntries: 1000,
        Registry:   reg,
    })
    defer r.Close()

    data, err := r.Read(context.Background(), src, "index.html")
    if err != nil {
        panic(err)
    }
    fmt.Println(string(data))
}

Sources

A Source pairs an io/fs.FS with a stable string ID used for cache namespacing. The ID must be non-empty and unique per logical filesystem.

// embed.FS — construct directly.
src := cfsread.Source{ID: "embed", FS: myEmbedFS}

// os.Root — confined to a directory, rejects traversal and symlink escapes.
src, closer, err := cfsread.NewRootSource("disk", "/var/data")
if err != nil {
    // handle error
}
defer closer.Close()

Cache configuration

r := cfsread.New(cfsread.Options{
    MaxBytes:   1 << 20,            // entries > 1 MB bypass the cache
    MaxEntries: 500,                // cap at 500 cached items
    MaxIdleAge: 10 * time.Minute,   // evict entries idle > 10 min
})

All bounds are optional. Zero means unlimited.

Invalidation

// Remove one cached entry.
r.Invalidate("disk", "config.yaml")

// Remove all entries for a filesystem.
r.InvalidateFS("disk")

Observability

Implement the Metrics and/or Logger interfaces to observe cache behaviour:

type Metrics interface {
    IncCacheHit()
    IncCacheMiss()
    IncCacheBypass()
    IncEviction(reason EvictionReason)
    ObserveDecompress(name string, inBytes, outBytes int64, d time.Duration)
    ObserveRead(d time.Duration)
}

type Logger interface {
    Debugf(format string, args ...any)
    Infof(format string, args ...any)
}

Pass them via Options.Metrics and Options.Logger. No-op implementations (NopMetrics, NopLogger) are used by default.

Plugin authoring

Implement the decompress.Decompressor interface and register it:

package zstd

type Decompressor struct{}

func (Decompressor) Name() string    { return "zstd" }
func (Decompressor) Magic() [][]byte { return [][]byte{{0x28, 0xB5, 0x2F, 0xFD}} }
func (Decompressor) Decompress(dst, src []byte) ([]byte, error) {
    // decompress src into dst, reusing dst's capacity if possible
    // ...
}

// Registration:
reg := decompress.NewRegistry()
if err := reg.Register(zstd.Decompressor{}); err != nil {
    // handle validation error (nil decompressor, empty/duplicate magic, etc.)
}

Requirements:

• Implementations must be safe for concurrent use.
• Magic must return at least one non-empty byte sequence.
• If two decompressors share a magic prefix, Register returns an error.
• Magic bytes are defensively copied at registration — mutation after Register has no effect.

Error handling

Read returns typed sentinel errors for validation failures, supporting errors.Is:

data, err := r.Read(ctx, src, name)
switch {
case errors.Is(err, cfsread.ErrReaderClosed):
    // reader was closed
case errors.Is(err, cfsread.ErrEmptySourceID):
    // Source.ID was empty
case errors.Is(err, cfsread.ErrNilSourceFS):
    // Source.FS was nil
}

The decompress package also exports sentinel errors for registration validation: ErrNilDecompressor, ErrNoMagic, ErrEmptyMagic, ErrDuplicateMagic.

Performance characteristics

• Zero-alloc cache hits: Once a file is cached, Read returns the cached byte slice without allocation. The caller must not mutate the returned slice.
• Singleflight coalescing: Concurrent reads of the same (source ID, path) pair share a single in-flight I/O + decompression call. Other goroutines await the result.
• Lazy eviction: Idle-age eviction is checked on access rather than via a background timer, avoiding goroutine overhead.
• Read buffer preallocation: Stat size is used to preallocate the read buffer, avoiding incremental growth on cache misses.

API overview

Type / Function	Description
Reader	Core cached reader (thread-safe)
Reader.Read(ctx, src, name)	Read a file, using cache and decompression
Reader.Invalidate(fsName, path)	Remove one cached entry
Reader.InvalidateFS(fsName)	Remove all entries for a filesystem
Reader.Close()	Mark reader as closed
Source	io/fs.FS + string ID pair
NewRootSource(id, dir)	Confined os.Root source
Options	Cache bounds, registry, metrics, logger
decompress.Registry	Magic-byte dispatch table
decompress.Decompressor	Plugin interface
lz4.Decompressor	Official LZ4 frame-format plugin
Metrics / Logger	Observation interfaces

Pre-compressing assets with cfsread-lz4

The cfsread-lz4 command recursively compresses files in a directory using LZ4 frame format, preserving file names. This is intended for use as a go:generate directive in consuming applications:

//go:generate go run github.com/lbe/cfsread/cmd/cfsread-lz4 ./assets

When the LZ4 decompressor is registered in the Reader, these files are transparently decompressed on read — no special handling needed in application code. Files already compressed in a known format (gzip, bzip2, xz, zstd, zip, etc.) are skipped automatically.

See cmd/cfsread-lz4/README.md for full documentation.

Architecture

See docs/ARCHITECTURE.md for package layout, data flow diagrams, and internal design.