StartLux-Prism

Fast, exact inference for StartLux-Decision models on NVIDIA GPUs. StartLux-Prism serves StartLux-Decision v1 and v1.1 AutoThink (4B, 9B, 27B) straight from their released GGUF files, behind the TypeSafe /v1/systemone API, on GPUs from Ampere to Blackwell, one or several. Like a prism that splits light into its spectrum, it turns each question into a calibrated probability for every option.

Live comparison · Installation · Quick start · Example · Models and GPUs · Serving guide · Benchmarks

_{StartLux-Decision-v1.1-27B-AutoThink on one H200: StartLux-Prism int8 against the official engine in FP8 (vLLM 0.31).}

_{StartLux-Decision-v1-27B on one H200: StartLux-Prism int8 against the official engine in FP8 (vLLM 0.31).}

The same StartLux-Decision v1.1 27B checkpoint answers 24 structured questions about one video in a single request: eight frames sampled from the clip (640x360), one H200, 8-bit on both sides. The official engine on vLLM 0.31 (FP8, its launcher's flags) took 0.72 s; StartLux-Prism int8 took 0.21 s (3.38x faster). Both engines ran on the same GPU, one after the other. The animation replays each at its measured speed: end-to-end HTTP time, median of 5 runs, model loading excluded. Watch the MP4. Video: Usman AbdulrasheedGambo / Pexels.

• Faster than the official engine. Serving the 27B on one H200, StartLux-Prism answers a short question 2.8× and a three-question request 5.3× faster than StartLux-Decision's own engine in FP8 (vLLM 0.31), with 1.17× its throughput from 64 clients. Eight H200s serve 238 requests/s, against 138 for the official engine in BF16.
• The official engine's accuracy. Prompts, read-outs, thoughts and decision rules are the model's own; only the forward pass is new. On StartLux-Decision v1's seven evaluation suites, int8 scores 92.94 and W4A8 93.07, against 92.97 for the official engine in BF16.
• Reproducible to the bit. int8 GEMMs sum exactly in int32, so a request receives the same probabilities whatever it is batched with, on any number of GPUs, from the prefix cache or not.
• All of v1.1. AutoThink thoughts are decoded on the GPU (95 tokens/s for one sequence, 3,832 for 64 at once); none_probability, allow_none and up to 8 images per request are supported.
• The released files as they are. Q8_0 files run as int8 (W8A8) and Q4_K_M files on their own 4-bit weights (W4A8), with no conversion step. In 4 bits the 27B runs on one 32 GB RTX 5090.
• One GPU or several. Data parallelism for throughput (3.92× on four H200s) and tensor parallelism (--tp 2 or 4) for latency.

With a CUDA 13 driver:

git clone https://github.com/StartLuxLabs/StartLux-Prism && cd StartLux-Prism
python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt     # PyTorch 2.14.1 for CUDA 13, Triton, nvcc 13.0
source scripts/cuda_env.sh          # nvcc from pip as CUDA_HOME (in every new shell)

With a CUDA 12 driver, replace the last two lines with PyTorch for CUDA 12.6 and a CUDA 12 toolkit's nvcc:

pip install -r requirements-cu12.txt
export CUDA_HOME=/usr/local/cuda    # a CUDA 12 toolkit (in every new shell)

A model directory holds the GGUF file and its release's folder files (config.json, decision_config.json, the tokenizer and the chat template). The v1 GGUF repositories include them; for v1.1, take them from the full release:

M=~/models/StartLux-Decision-v1.1-27B-AutoThink-GGUF
hf download startlux-models/StartLux-Decision-v1.1-27B-AutoThink-GGUF StartLux-Decision-v1.1-27B-AutoThink-Q8_0.gguf --local-dir $M
hf download startlux-models/StartLux-Decision-v1.1-27B-AutoThink --exclude "*.safetensors" --local-dir $M   # tokenizer and config

Add the mmproj-*.gguf file to read images. The model weights are licensed CC BY-NC 4.0 (non-commercial).

Start a server:

G=$M/StartLux-Decision-v1.1-27B-AutoThink-Q8_0.gguf
python -m startlux_prism.server --gguf $G --model-dir $M --port 8092    # ready when it prints "Ready"

The server prints each loading step, then Ready, its endpoints and a request to try (what a start prints). The first start measures the int8 activation ranges once and keeps them in ~/.cache/startlux-prism/calib.

The same flags apply, and every answer is bit-identical to the single-GPU one:

python -m startlux_prism.dp --gpus 0,1,2,3,4,5,6,7 --gguf $G --model-dir $M --port 8092   # data parallel: most throughput
python -m startlux_prism.server --tp 2 --gpus 0,1 --gguf $G --model-dir $M --port 8092    # tensor parallel: less latency
python -m startlux_prism.dp --gpus 0,1,2,3 --tp 2 --gguf $G --model-dir $M --port 8092    # both

The server has no authentication of its own: serve other machines (--host 0.0.0.0) only behind your own access control. To keep the compiled kernels and the calibration across restarts, or to run the server under systemd, see Running as a service.

With the server running, ask one question:

curl -s localhost:8092/v1/systemone -H 'Content-Type: application/json' \
  -d '{"state": "I was charged twice and want a refund.", "questions": {"intent": {"type": "choice", "instructions": "Choose the customer intent.", "criteria": {"billing": "A payment or refund issue", "technical": "A malfunction or setup issue", "other": "Another request"}}}}'

The response holds answers.intent.choice, the most likely label, and answers.intent.probabilities, one per label.

Several questions about the same evidence go in one request and are answered in one forward pass. Ask two questions about one ticket:

curl -s localhost:8092/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"ticket": "I was charged twice for order #4411 and the app still shows it as unpaid."},
  "questions": {
    "team": {"type": "choice", "instructions": "Which team should handle this ticket?",
             "criteria": {"billing": "Payments and refunds", "shipping": "Delivery", "technical": "App and login"}},
    "urgent": {"type": "noul", "instructions": "Should this ticket be answered today?"}}}'

{"answers": {"team": {"type": "choice", "choice": "billing", "confidence": 0.993, "probabilities": {...},
                      "none_probability": 0.002},
             "urgent": {"type": "noul", "noul": 0.749}},
 "usage": {"input_tokens": 191, "output_tokens": 0}, ...}

Both first read-outs reached the default gate (0.6), so the model answered without thinking. With "gate": 0.9, the setting of the published numbers, urgent (0.749) falls below it: the model thinks for 29 tokens and answers 0.855. none_probability is the probability that no listed option fits; the serving guide explains how to act on it.

Ask about a video: sample eight frames, then send them as images (v1.1 with its mmproj file, up to 8 per request). A string state places them at its <image> marks:

ffmpeg -i clip.mp4 -vf "fps=8/21,scale=640:360" frame%d.jpg    # eight frames from a 21-second clip, at 640x360

import base64, json, urllib.request

frames = ["data:image/jpeg;base64," + base64.b64encode(open(f"frame{i}.jpg", "rb").read()).decode()
          for i in range(1, 9)]
body = {"state": "Eight frames sampled in order from one video of a work site: " + " ".join(["<image>"] * 8),
        "images": frames,
        "think_mode": "never",
        "questions": {
            "vehicle": {"type": "choice", "instructions": "Which vehicle is used?",
                        "criteria": {"forklift": "A sit-down forklift", "order picker": "A stand-up order picker",
                                     "none": "No vehicle"}},
            "pedestrian_near": {"type": "noul",
                                "instructions": "Is a person on foot within a few meters of the vehicle?"},
            "risk": {"type": "score", "instructions": "How risky is the situation overall?",
                     "criteria": ["Low", "Medium", "High"]}}}
req = urllib.request.Request("http://localhost:8092/v1/systemone", json.dumps(body).encode(),
                             {"Content-Type": "application/json"})
print(json.dumps(json.load(urllib.request.urlopen(req))["answers"], indent=1))

The live comparison sends this kind of request, with 24 questions.

--precision bf16 runs the same weights dequantized, on cuBLAS, as the reference for parity tests.

Data parallelism runs every precision; tensor parallelism runs the int8 path on 2 or 4 GPUs (measured on H200s). The models and files measured on each GPU: Models, GPUs and precisions.

Endpoints: POST /v1/systemone (also /v1/decisions), GET /v1/models, GET /health and GET /stats.

Every field, flag and model folder: serving guide.

The 27B on one H200 per server, against the official engine (Startlux-decide-inference on vLLM 0.31) on the same requests. Latency is the mean over requests sent one at a time.

StartLux-Decision-v1.1-27B-AutoThink

StartLux-Decision-v1-27B (v1 answers in one pass)

Under load both engines are bound by the tensor cores, which run int8 and FP8 at the same rate, so the lead narrows to 17% on v1.1 and 22% on v1. On several H200s, v1.1 serves 30.9 read-outs per second on one GPU, 121.1 on four and 237.8 on eight (the official engine on eight: 137.5).

Against every other engine compared on v1 (SGLang, vLLM, llama.cpp, StartLux-Decision's PyTorch server, decider and JevK5), StartLux-Prism has the lowest latency (1.6-2.3× below the next) and 1.6-2.4× their throughput. Methods, every engine and the raw numbers: benchmarks.

StartLux-Decision-v1.1-27B-AutoThink with thoughts: StartLux-Prism int8 from the Q8_0 file, and W4A8 from the Q4_K_M file on one RTX 5090, against the official engine on the BF16 release.

_{StartLux-Decision-v1.1-27B-AutoThink with thoughts: accuracy.}

On StartLux-Decision v1's own evaluation (12,351 decisions), int8 averages 91.14 against the reference's 91.15 on the 4B, and 90.80 against 91.06 on the 9B. About 1% of the decisions change, all near ties, and calibration does not move. Every suite: benchmarks.

• One forward pass per batch, as CUDA graphs. A request's questions branch off their shared evidence inside the graph, so the evidence is computed once.
• A prefix cache keeps evidence across requests: a follow-up question takes 5.4 ms instead of 54.
• int8 tensor cores with exact int32 sums, so an answer does not depend on its batch, the number of GPUs or the cache.
• Kernels chosen per GPU from measured tables. On Hopper the int8 GEMMs run on warpgroup MMA at 1,240-1,420 TOPS.
• Thoughts decoded on the GPU in CUDA graphs, between prefill batches, seeded per question.

• StartLux-Decision: the models. StartLux-Prism vendors its prompt format (startlux_prism/jevfmt.py) and int8 calibration requests (calibration.py), and the thought loop detector of its v1.1 package (loops.py).

StartLux-Prism's own code does not have a license yet; it will be chosen by the owner. The files vendored from StartLux-Decision (startlux_prism/jevfmt.py, startlux_prism/calibration.py, startlux_prism/loops.py) are under Apache-2.0 (LICENSES/StartLux-Decision-Apache-2.0.txt). Model weights are not part of this repository; StartLux-Decision's weights are CC BY-NC 4.0 (non-commercial).