Reflex

A GGUF-native Rust and CUDA inference engine built for cold starts: the time from launching a process to its first result, not sustained server throughput. Every CUDA kernel is compiled ahead of time by nvcc and shipped inside the binary, so nothing is JIT-compiled when the process starts.

~483 ms cold start, p50: reflex system1 from process start to its scored result (the engine's own timer), Tesla T4, Qwen3-0.6B-Q4_K_M, n=10. Measured from outside the process, including the ~110 ms the OS spends launching it, that is ~0.6 s wall clock. Phase breakdown. Measured with f32 weights, before f16 weight storage became the default. A same-session T4 comparison of the two weight types puts f16 7% faster here, with half the VRAM and 36–42% faster decode (f16 weight storage).

Architectural boundary. Reflex is a local execution engine for single-tenant decisions: one request at a time, no scheduler, no batching, no network server inside the engine. HTTP is a separate sidecar (sidecar/openai-adapter) over local IPC. See Non-goals before proposing otherwise.

Serverless GPU platforms bill per second of wall clock per invocation, so for bursty, single-shot work (a tool call, a classification, an extraction) the time a worker takes to become useful is the bill. Engines built for steady-state throughput pay for that with long startups: vLLM captures CUDA graphs and compiles at launch. llama.cpp, like Reflex, ships precompiled kernels and is the honest comparison point.

On a real Runpod serverless endpoint, Reflex's own load took ~1.6 s of a ~42-44 s cold invocation; the rest is the platform provisioning the GPU and container, which no engine avoids. Runpod's official vLLM worker took ~150 s on the same GPU tier the same day. Against an official llama.cpp server image on the same platform, Reflex's engine load was ~0.5 s vs. ~0.95 s on an L4, but end-to-end the platform's own variance was far larger than that gap (details). The full argument and measurements are in docs/benchmarks.md.

Requires the CUDA toolkit (nvcc on PATH, or CUDA_PATH/CUDA_HOME set) and an NVIDIA GPU. REFLEX_SKIP_CUDA=1 cargo build type-checks without CUDA (nothing will run).

git clone https://github.com/lateos-ai/reflex.git
cd reflex
cargo build --release

# Prove the AOT kernel pipeline works on your GPU:
cargo run --release --bin reflex -- smoke

# Score candidate answers in one forward pass:
cargo run --release --bin reflex -- system1 model.gguf "The capital of France is" \
  --candidate " Paris" --candidate " London"

# Generate text:
cargo run --release --bin reflex -- generate model.gguf "Once upon a time" --max-tokens 32

reflex <subcommand> with no arguments prints that subcommand's usage. All subcommands, build options and output formats: docs/reference.md.

No GPU of your own? Reflex is listed on the Runpod Hub: deploy it as a serverless endpoint in one click, no build required (worker docs).

Each architecture counted as supported only after its generated tokens matched an independent implementation (llama.cpp, or a CPU reference) on real hardware.

• GGUF only. Convert other checkpoints with llama.cpp's convert_hf_to_gguf.py.
• Weights are held as f16 on the GPU by default, dequantized once at load: VRAM is about 2 bytes per parameter whatever the file's quantization. --weights f32 (or REFLEX_WEIGHTS=f32) keeps the exact f32 reference mode, about 4 bytes per parameter. Only matrix weights change type; norms, MoE routers, activations and the KV cache are f32 in both modes. reflex check defaults to f32.
• Context: up to 65,535 positions per sequence (prompt + generated tokens + any imported KV cache); longer requests fail with a clear context_overflow error.
• system1 works on every architecture; on the Qwen3.5 hybrid models each candidate must be a single token.
• Models too big for f16: REFLEX_QUANT_RESIDENT=1 keeps Q4_K/Q6_K matrix weights quantized on the GPU instead (dense path and Kolibri-1), and Q8_0 ones on Gemma 4. Kolibri-1's 47.5 GB Q4_K_M then fits a 48 GB card, and so does Gemma 4 31B Q8_0 (37.9 GB peak at 2,641 tokens). On Gemma 4 it also speeds up cold start and decode, but short warm prompts are slower than with f16 weights. Adding REFLEX_LAZY_EXPERTS=1 uploads each expert only when the router first picks it: a short prompt's cold first token on an RTX A6000 goes from 8.6 s to 2.3 s. See quantized-resident weights and Kolibri-1.
• DeepSeek's Q-LoRA query compression and MTP heads are not supported.

Cold starts measured from outside the process, Tesla T4, Qwen3-0.6B-Q4_K_M, unless noted. The engine-vs-engine rows time reflex generate to its first token (0.83 s p50, n=30), the metric every engine shares; the headline above times system1 scoring. Every Reflex figure here was measured with f32 weights, before f16 became the default. In a same-session comparison of the two weight types, f16's first token was 8% sooner (f16 weight storage).

Methodology, per-run numbers and every caveat: docs/benchmarks.md.

Energy. Measured from outside the process with the GPU's idle draw subtracted, a cold reflex system1 on a T4 costs ~5.4 J. Most of the ~28 J the GPU draws over that window is idle power. Details.

For a scale-to-zero endpoint, start with the serverless deployment guide — one page, Runpod first. The job Reflex is built for is a single cold decision (POST /v1/classify), not a warm chat pool; see the sidecar's classify section. The table below is the long form.

The Python bindings (--features python) do not currently build.

These are permanent, not a backlog:

• batch_size is always 1. No request queue, no continuous batching, no PagedAttention-style allocation. Run more processes to serve more requests.
• No network server in the engine, ever. HTTP lives in the separate sidecar; the engine itself offers only sequential local IPC and in-process bindings.
• No multi-tenant LoRA router, no internal KV-cache store. Adapters load once at startup; where a KV-cache file lives is the caller's business.
• No scheduler or autoscaling. The platform (Kubernetes, a serverless provider, an autoscaling group) is the orchestrator; the engine stays a single-shot process.

The reasoning, and what the engine may grow instead: docs/DEVELOPMENT.md.

• docs/reference.md: subcommands, build features and kernel modes, --json output, error categories, Docker, Kubernetes
• docs/benchmarks.md: every measurement, with method and caveats
• docs/DEVELOPMENT.md: building, testing, and the rules for changes
• docs/gpu-ci.md: the nightly GPU workflow
• CHANGELOG.md: what has been built, in order

MIT; see LICENSE. Contributions need a signed Contributor License Agreement; see CONTRIBUTING.md.