scribe topic log

scribe is an embeddable, append-only topic queue for Go — a single-node, disk-backed log with per-consumer offsets that operates as a stand-alone. No external dependency; only the Go standard library.

Documentation

• Executive summary — what scribe is and why it matters
• User guide — configuration, API, and operational notes
• Architecture — design, data flow, trade-offs, and limitations
• Operational plan — capacity, sizing, and tuning for high-volume fan-in logging
• Examples — direct embedding, an HTTP server, a Unix-socket server with the binary wire encoding, and an MCP server that gives an AI assistant a durable log

Features

• sharded, rotating binary log segments on disk (CRC-checked frames)
• per-topic sequential record numbering with per-shard parallel writers
• buffered, batched fsync writes; memory-mapped, offset-indexed reads
• prefix-consistent reads: a record is never visible while an earlier sequence is still buffered, so committing the highest sequence read is always safe (Durable marks the boundary; gaps below it are permanent)
• replayable reads from any offset; zero-copy Scan / ScanAll variants
• consumer offset commits for resume/replay workflows, with RegisterConsumer retention protection and observable data loss (TopicStat.Dropped)
• retention by whole-segment deletion, plus per-topic and global byte caps — or Retention: -1 for a never-expiring archival log
• single-writer safety: an exclusive directory lock (linux/darwin) plus a ReadOnly mode for inspecting a live directory without mutating it
• an opt-in fixed-width record format (PayloadWidth) with O(1) cold-start load
• a compact binary wire encoding (Record.MarshalBinary / ParseRecord) for sending records over a network

Usage

import (
    "time"

    "github.com/netstar-labs/scribe"
)

q, err := scribe.New(scribe.Config{
    Dir:       "./data/queue",
    Shards:    4,
    Retention: 24 * time.Hour,
})
if err != nil {
    panic(err)
}
defer q.Close() // flushes buffered records and stops background maintenance

rec, err := q.Append("orders", []byte("hello")) // rec.Seq is the assigned offset
if err != nil {
    panic(err)
}

// Appends are buffered; flush before a read-your-writes read
// (or rely on the background flush / ReadWait).
_ = q.Flush()

recs, err := q.Read("orders", 0, 10) // records with seq >= 0, up to 10
if err != nil {
    panic(err)
}

_ = q.Commit("orders", "worker-a", rec.Seq)
offset, _ := q.Offset("orders", "worker-a")

_ = recs
_ = offset

CLI

cmd/scribe manages and inspects a data directory: list topics, read/tail/dump records, append, track and set consumer offsets, health-check, and apply retention on demand. The on-disk format (variable vs fixed-width) is auto-detected, and inspection commands never delete records — only the explicit prune command applies retention:

GOWORK=off go run ./cmd/scribe -dir ./data/queue topics
GOWORK=off go run ./cmd/scribe -dir ./data/queue append orders "hello" "world"
GOWORK=off go run ./cmd/scribe -dir ./data/queue tail orders 10
GOWORK=off go run ./cmd/scribe -dir ./data/queue read orders 1 100
GOWORK=off go run ./cmd/scribe -dir ./data/queue cat orders
GOWORK=off go run ./cmd/scribe -dir ./data/queue commit orders worker-a 42
GOWORK=off go run ./cmd/scribe -dir ./data/queue register orders worker-a
GOWORK=off go run ./cmd/scribe -dir ./data/queue unregister orders worker-a
GOWORK=off go run ./cmd/scribe -dir ./data/queue offsets
GOWORK=off go run ./cmd/scribe -dir ./data/queue check
GOWORK=off go run ./cmd/scribe -dir ./data/queue prune -retention 48h

Inspection commands open read-only and are safe against a directory a live process owns; management commands (append, commit, register, unregister, prune) take the exclusive directory lock and fail cleanly if a live process holds it.

Test

GOWORK=off go test ./...
GOWORK=off go test -race ./...
GOWORK=off go test -run '^$' -bench . -benchmem .

License

Copyright (C) 2026 zxdev. Licensed under the GNU General Public License v3.0 or later — see LICENSE. The internal/mmap package is derived from golang.org/x/exp/mmap and retains its BSD-style license (internal/mmap/LICENSE).