Mini-Rockset

A high-performance, high-throughput data ingestion and columnar storage pipeline built with Go, Protocol Buffers, and C++.

Features

• Fast Ingestion: 10k-100k records/sec via Go HTTP service
• Strict Typing: Protocol Buffers for language-agnostic data contracts
• Durability: NATS JetStream provides at-least-once delivery with disk persistence
• High Performance: C++ columnar storage engine for efficient analytical queries
• Backpressure: Automatic load shedding when ingestion exceeds downstream capacity
• Scalability: Stateless ingestion service, horizontal scaling via NATS and storage replication

Architecture

Clients
  ↓
Go Ingestion Service (Worker Pool, Backpressure)
  ↓
NATS JetStream (Buffer, Durability)
  ↓
C++ Storage Engine (Columnar, Queryable)

For detailed architecture, see docs/ARCHITECTURE.md.

Quick Start

Prerequisites

• Go 1.22+
• Protocol Buffers (protoc) and buf
• C++17 compatible compiler (for Phase 4)
• Docker & Docker Compose (for NATS)

Setup & Development

See docs/DEV_SETUP.md for local development instructions.

Phase Progress

• Phase 1: Protobuf Schema Definition ← You are here
• Phase 2: High-Throughput Go Ingestion Service
• Phase 3: NATS JetStream Integration
• Phase 4: C++ Storage Engine

Project Structure

mini-Rockset/
├── proto/                      # Protobuf definitions
├── gen/{go,cpp}/              # Generated code (Protobuf → Go/C++)
├── cmd/ingestion-service/     # Go ingestion service entry point
├── internal/                  # Go service packages
│   ├── server/               # HTTP handlers
│   ├── worker/               # Worker pool
│   ├── publisher/            # Message publishing
│   ├── consumer/             # Message consuming
│   ├── backpressure/         # Backpressure logic
│   ├── config/               # Configuration
│   └── metrics/              # Observability
├── cpp/                       # C++ storage engine
│   ├── src/                  # Source files
│   ├── include/              # Headers
│   ├── tests/                # Unit tests
│   └── CMakeLists.txt        # Build config
├── docs/                      # Documentation
│   ├── ARCHITECTURE.md
│   ├── SCHEMA.md
│   ├── DEV_SETUP.md
│   └── DEPLOYMENT.md
├── docker-compose.yml        # NATS local dev environment
├── buf.yaml                  # Protobuf build config
├── buf.gen.yaml             # Protobuf code generation rules
├── go.mod                    # Go module file
└── Makefile                  # Build commands (optional)

Data Model

The core data structure is a Record:

message Record {
  string id = 1;                          // Unique identifier
  int64 timestamp = 2;                    // Unix timestamp (ms)
  bytes payload = 3;                      // Opaque data
  map<string, string> metadata = 4;       // Tags for filtering
}

Records are sent in batches via HTTP POST to /v1/ingest.

For full schema details, see docs/SCHEMA.md.

Build & Test

Generate Code (Phase 1)

# Install buf if not present
brew install buf

# Lint Protobuf definitions
buf lint proto/

# Generate Go and C++ code
buf generate proto/

Build Go Service (Phase 2+)

go build -o bin/ingestion-service ./cmd/ingestion-service

Build C++ Engine (Phase 4+)

cd cpp
mkdir build && cd build
cmake ..
make

Deployment

See docs/DEPLOYMENT.md for production deployment guide.

Documentation

• Architecture — System design, layers, data flow
• Schema — Data contract, API specs, constraints
• Development Setup — Local setup and testing
• Deployment — Production deployment

Contributing

1. Protobuf changes: Run buf lint and buf generate before committing
2. Go changes: Run go fmt, go vet, and tests before committing
3. C++ changes: Compile with no warnings, run tests

License

(Add your license here)

Status

Current Phase: Phase 1 - Protobuf Schema Definition (In Progress)

Timeline:

• Phase 1: Schema definition and code generation ← Active
• Phase 2: Go ingestion service + worker pool
• Phase 3: NATS JetStream integration
• Phase 4: C++ storage engine

Contact

For questions or issues, reach out via issues or PRs.