Rectifiers

Cache-aware routing and orchestration layer for multi-GPU LLM inference.

中文文档 (Chinese)

Rectifiers sits between your API gateway and vLLM/SGLang inference workers, providing cache-aware request routing (query the Director for KV-cache prefix hits before dispatching) and dynamic P:D ratio auto-tuning (Orchestrator scales prefill/decode Deployments via K8s). Built on top of RDMAS (One-Sided RDMA shared storage) and LMCache (LLM KV-cache library).

---

Design Principles

1. Zero-CPU data plane — All KV-cache I/O uses RDMAS One-Sided RDMA READ/WRITE/CAS. RDMAS server CPU never touches the hot path. Rectifiers only maintains a cache-location index.
2. Index-data separation — Director's PrefixIndex is an in-memory index. KV-cache data lives in RDMAS nodes. The index never stores data.
3. Cache-aware scheduling — Router queries Director before every request to find the RDMAS node with the most prefix hits, then dispatches to the worker nearest that node. Falls back to least-loaded on cold start.
4. Two-phase write reporting — Connector records submit_batch_set metadata but only calls report_store in drain_completions when comp.ok == true. No phantom cache hits.
5. Progressive integration — Director, Router, and Orchestrator deploy as standalone Rust binaries or as a single combined binary. Connector adds Director reporting via an optional director feature flag.

vs Mooncake

Dimension	Mooncake	Rectifiers
Data plane	Transfer Engine (CPU-coordinated)	RDMAS One-Sided RDMA (zero CPU)
Index update	ZMQ pull (Go process + etcd)	Connector drain_completions push
Index latency	ZMQ network RTT	<1µs in-process / <100µs gRPC local
P:D auto-tuning	None	Orchestrator hysteresis + K8s scaling
Language stack	Go + C++	Pure Rust

---

Quick Start

Prerequisites

• Rust 1.80+
• protoc (protobuf-compiler)
• Optional: K8s cluster (Orchestrator), RDMA NIC + libibverbs (RDMAS), python3-dev (Connector)

# Install protoc
sudo apt install protobuf-compiler   # Debian/Ubuntu
sudo dnf install protobuf-compiler   # Fedora
brew install protobuf                # macOS

Build & Test

git clone https://github.com/ipconfiger/rectifiers
cd rectifiers

# Build all crates
cargo build --workspace --release

# Run all tests (81 tests)
cargo test --workspace --exclude rectifiers-connector
cargo test -p rectifiers-connector --no-default-features --features director

Run Services

# Combined binary (Director + Router, recommended)
cargo run -p rectifiers --release              # HTTP :8080 + gRPC :9200

# Standalone (development)
cargo run -p rdmas-director --release          # gRPC :9200
cargo run -p rdmas-router --release            # HTTP :8080
cargo run -p rdmas-orchestrator --release      # gRPC :9201

Docker

docker build -t rectifiers -f docker/Dockerfile.rectifiers .
docker compose -f docker/docker-compose.yml up -d

---

Architecture

Client → API Gateway (Nginx/Traefik) → Rectifiers Router (:8080)
                                            │ gRPC query
                                            ▼
                                      Director (:9200)
                                      PrefixIndex + InstanceRegistry
                                            │
                                   ┌────────┴────────┐
                                   │                  │
                              Orchestrator           Connector
                              (:9201, K8s scale)     (PyO3, in vLLM)
                                   │                  │
                                   ▼                  ▼
                              vLLM Workers          RDMAS Storage
                              (Prefill + Decode)    (HugePage KV cache)

Request Lifecycle

1. Client sends prompt → API Gateway → Router
2. Router tokenizes prompt → computes block hashes → queries Director
3. Director returns ranked hits (node, matched_blocks, nearby_workers)
4. Router selects best worker (tier × hit count × load) → dispatches
5. Worker (Prefill): GPU compute → LMCache writes KV cache to RDMAS via One-Sided RDMA WRITE
6. Connector (in drain_completions): report_store(block_hashes) → Director updates PrefixIndex
7. Worker (Decode): LMCache loads KV cache from RDMAS via One-Sided RDMA READ → generates tokens
8. SSE token stream → Router → API Gateway → Client

---

Crates

Crate	Lang	Port	Description
rdmas-director	Rust	gRPC :9200	Global KV-cache index + worker registry (8 RPCs)
rdmas-router	Rust	HTTP :8080	Cache-aware request routing + SSE proxy
rdmas-orchestrator	Rust	gRPC :9201	P:D ratio auto-tuning via K8s Deployment scaling
rectifiers (combined)	Rust	:8080+:9200	Director + Router in a single process (prod recommended)
rectifiers-connector	Rust/PyO3	—	LMCache native_plugin with optional Director reporting
router-py	Python	HTTP	FastAPI prototype for quick validation

---

GPUStack Integration

Rectifiers integrates with GPUStack with zero code changes:

1. Deploy Rectifiers → kubectl apply -f k8s/
2. Point GPUStack AI Gateway upstream to Rectifiers Router (:8080)
3. Add --lmcache-config to vLLM backend parameters with director_addr set
4. Build custom vLLM image with liblmcache_rdma_connector.so
5. Enable Orchestrator for P:D auto-tuning

See GPUStack Integration Guide and Deployment Guide for full details.

---

LMCache Configuration

Add to vLLM startup args:

--lmcache-config '{
  "type": "native_plugin",
  "module_path": "lmcache_rdma_connector",
  "class_name": "RDMANativeConnector",
  "adapter_params": {
    "device": "mlx5_0",
    "server": "10.0.0.1:9400",
    "num_workers": 4,
    "director_addr": "rectifiers-director:9200",
    "node_id": "rdmas-0",
    "tenant_id": "default",
    "model_name": "llama-70b",
    "block_size": 16
  }
}'

adapter_params Fields

Field	Required	Default	Description
device	Yes	—	RDMA device name, e.g. mlx5_0
server	Yes	—	RDMAS storage node host:9400
num_workers	No	4	RDMA worker threads
director_addr	No	—	Set to enable Rectifiers cache reporting
node_id	Yes*	—	RDMAS node ID for this worker
tenant_id	Yes*	—	Tenant isolation key
model_name	Yes*	—	Must match Router config
block_size	Yes*	16	Must match Router config
instance_id	No	auto	Worker instance ID
role	No	both	prefill / decode / both

* Required when director_addr is set.

Connector Auto-Behavior

When director_addr is configured, the Connector automatically:

• Register — registers this worker with the Director on startup
• Heartbeat — sends heartbeat every 10s
• ReportStore — reports (node_id, block_hashes) in drain_completions only on successful RDMA writes
• ReportRemove — reports removals on successful deletes
• Deregister — deregisters on shutdown

---

RDMAS Storage Backend

Rectifiers does not store KV-cache data — data lives in RDMAS nodes. The Connector writes to RDMAS via One-Sided RDMA and reports block locations to the Director.

Deploy RDMAS Nodes

# Allocate HugePages (512 GB)
echo 262144 > /proc/sys/vm/nr_hugepages

# Start RDMAS Server
cd /path/to/rdmas && cargo run --release
# ControlPlane gRPC on :9400

---

Deployment

K8s (Production)

kubectl apply -f k8s/rectifiers.yaml        # Combined binary (replicas=1)
kubectl apply -f k8s/orchestrator.yaml      # Orchestrator + RBAC

Docker Compose (Local)

docker compose -f docker/docker-compose.yml up -d
# Rectifiers (:8080+:9200) + Orchestrator (:9201) + mock vLLM (:8010)

---

Development

# Tests
cargo test --workspace --exclude rectifiers-connector
cargo test -p rectifiers-connector --no-default-features --features director

# Lint
cargo clippy --workspace -- -D warnings
cargo fmt --all -- --check

# Benchmarks
cargo bench --workspace

Documentation

Document	Description
Architecture Design	15-chapter specification: proto, data model, routing algorithm
Development Plan	8-phase task breakdown
Deployment Guide	Full deployment guide with verification
GPUStack Integration	Zero-code-change GPUStack integration
Review Report	Multi-oracle audit findings

---

License

MIT. See LICENSE for details.