Slabs: Simpler Flat-Buffers like Serialization for Go

Slabs is a Go library and code generator that brings flat, contiguous memory representation for suitable Go structs with zero-copy readers/writers, type descriptors, and version-friendly encoding/decoding — inspired by FlatBuffers, but simplified for idiomatic Go development.

It’s designed for situations where you need:

• Data in fixed size units of memory like disk blocks
• Fast serialization/deserialization with minimal allocations
• Direct memory access without intermediate object building
• Type safety with generated code
• Schema evolution that supports adding/removing fields over time without breaking compatibility
• Portability across different OSes and architectures

Features

• Flat Memory Representation

  Store structs in contiguous byte slices for efficient I/O and caching.

• Code Generation for Structs

  From your Go type definitions, generate:

  • Reader types for zero-copy, read-only access to fields
  • Writer types to build serializable objects in flat memory
  • Type descriptors with a unique signature for each type

• Type Descriptor Registry

  Map between type signatures and runtime type metadata for decoding.

• Encoding/Decoding

  Built-in serialization that allows evolving your struct definition without breaking old data.

• Schema Evolution

  Add or reorder fields without requiring a complete rewrite of existing serialized data.

Restrictions on Struct Fields

To represents structs in flat memory, code generator enforces following restrictions on the struct type definitions.

We must place restrictions on struct type definitions so that struct objects can be represented in flat memory.

• No int/uint type fields for portability

  Byte size for these types is platform dependent, so they make the slabs non-portable.

• No string, pointer, map, etc. fields

  These types of fields cannot be represented in flat memory.

• At most ONE slice field for dynamic needs

  One slice field is allowed in top-level structs to accommodate dynamic content. However, such structs with slice field cannot be used as fields of other structs anymore.

Getting Started

1. Add slabgen dependency to your go.mod file

go get github.com/visvasity/slabgen@latest

2. Define Your Struct

// superblock.go
package example

import "crypto/sha256"

type LSN int64

type BlockType uint16

type BlockHeader struct {
  Checksum [sha256.Size]byte

  BlockLSN LSN

  BlockTypeID BlockType
}

type SuperBlock struct {
  Header BlockHeader

  ...
}

3. Generate Reader/Writer types

go run github.com/visvasity/slabgen -indir=. -outdir=. -outpkg=example SuperBlock

This generates:

• superblock.slab.go -- Reader/Writer types, encoding/decoding logic
• Type descriptors for registration code

Example Code

package main

import (
    "fmt"
    "bytes"
    "github.com/foo/example"
)

func main() {
    bs := make([]byte, 4*1024)
    block := example.NewSuperBlock(bs)

    reader := block.Header()
    writer := reader.Writer()

    writer.SetLSN(0)
    writer.SetBlockType(0)
    writer.SetChecksum(sha256.Sum(bs))

    fmt.Println("LSN:", reader.LSN())
    fmt.Println("Type:", reader.BlockType())
    ...
}

Why Not FlatBuffers?

FlatBuffers is powerful, but:

• It requires an external schema language and compiler
• It doesn’t integrate naturally with Go type definitions
• It can be overkill for many Go projects

Slabs stay entirely within Go, so:

• No .fbs files — your Go structs are the schema
• More natural Go code with named types and aliases

Roadmap

• Benchmarks vs. FlatBuffers/JSON/protobuf
• Add builtin support for time.Time, etc. standard types
• Optional compression support