Silk

A Redis-like in-memory key-value store written in Go, with AOF (Append-Only File) persistence.

Features

• String-only in-memory key-value store with SET, GET, DEL, EXISTS
• AOF persistence that appends write commands to disk
• AOF replay to rebuild in-memory state on startup
• Automatic AOF rewrite/compaction into base snapshots
• Manifest-tracked file state for safe crash recovery
• RESP command encoding and parsing for persisted commands
• TCP server (planned)

System Architecture

┌─────────────────────────────────────────────────────────┐
│                        Client                           │
│                  (RESP over TCP) [planned]              │
└─────────────────────────┬───────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────┐
│                     TCP Server                          │
│            Accept → parse RESP → dispatch               │
│                      [planned]                          │
└─────────────────────────┬───────────────────────────────┘
                          │ []string{cmd, args...}
┌─────────────────────────▼───────────────────────────────┐
│                       Store                             │
│              internal/store/store.go                    │
│                                                         │
│   map[string]string  protected by sync.RWMutex          │
│                                                         │
│   SET  ──► mutate map  ──► send "SET k v" to LogCh      │
│   DEL  ──► mutate map  ──► send "DEL k"   to LogCh      │
│   GET  ──► read map                                     │
│   EXISTS ► read map                                     │
└─────────────────────────┬───────────────────────────────┘
                          │ chan<- string (LogCh)
┌─────────────────────────▼───────────────────────────────┐
│                    AOF Manager                          │
│                internal/aof/aof.go                      │
│                                                         │
│  WriteLog goroutine                                     │
│  ┌──────────────────────────────────────────────────┐   │
│  │ for command := range commandChannel              │   │
│  │   if incrFile.size >= 10MB ──► Rewrite()         │   │
│  │   encode command as RESP                         │   │
│  │   Append(command) ──► write to current incr file │   │
│  └──────────────────────────────────────────────────┘   │
│                                                         │
│  Rewrite()                                              │
│  ┌──────────────────────────────────────────────────┐   │
│  │ 1. increment sequence index                      │   │
│  │ 2. open new incr file (index N)                  │   │
│  │ 3. snapshot store data  ──► write to base file   │   │
│  │ 4. update manifest (Reset to base + incr at N)   │   │
│  └──────────────────────────────────────────────────┘   │
│                                                         │
│  Replay() — called once on startup                      │
│  ┌──────────────────────────────────────────────────┐   │
│  │ 1. read manifest to find max sequence N          │   │
│  │ 2. load base file at N  ──► replay SET commands  │   │
│  │ 3. load incr file at N  ──► replay SET/DEL cmds  │   │
│  └──────────────────────────────────────────────────┘   │
└─────────────────────────┬───────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────┐
│                     Manifest                            │
│             internal/manifest/manifest.go               │
│                                                         │
│   appendonly-dir/appendonly.aof.manifest                │
│                                                         │
│   Tracks active AOF files as ordered entries:           │
│   file appendonly.2.base.aof seq 2 type b               │
│   file appendonly.2.incr.aof seq 2 type i               │
│                                                         │
│   Add()   — append one entry                            │
│   Reset() — truncate and rewrite all entries            │
└─────────────────────────┬───────────────────────────────┘
                          │
┌─────────────────────────▼───────────────────────────────┐
│                    appendonly-dir/                       │
│                                                         │
│   appendonly.aof.manifest                               │
│   appendonly.N.base.aof   ← full snapshot at rewrite   │
│   appendonly.N.incr.aof   ← commands since snapshot    │
└─────────────────────────────────────────────────────────┘

AOF File Lifecycle

Startup
  └── NewAOFManager()
        ├── load manifest
        ├── derive current sequence index from manifest
        ├── open appendonly.<N>.incr.aof
        └── register incr entry in manifest if missing

Normal operation
  └── each write command
        └── Store sends "SET k v" or "DEL k" to LogCh
              └── WriteLog encodes and appends RESP to appendonly.<N>.incr.aof

Rewrite (triggered when incr file >= 10MB)
  ├── increment N
  ├── snapshot all store data → appendonly.<N>.base.aof
  ├── open appendonly.<N>.incr.aof  (new commands go here)
  └── manifest.Reset([base@N, incr@N])  (old files superseded)

Recovery (on next startup)
  ├── Replay(base@N)  → rebuild full string state
  └── Replay(incr@N) → apply commands since last snapshot

Package Layout

silk/
├── cmd/
│   └── main.go              entry point
└── internal/
    ├── store/
    │   └── store.go         string-only in-memory map + write-log channel
    ├── aof/
    │   └── aof.go           AOF manager (write, replay, rewrite)
    ├── resp/
    │   └── resp.go          RESP command parser
    └── manifest/
        └── manifest.go      manifest load, add, reset

Planned

• TCP server with per-connection goroutines
• Configurable rewrite threshold
• KEYS, FLUSH, TTL commands