Repository navigation
SGLang Backend
A durable disk tier underneath SGLang's RadixAttention.
| Status | ✅ Works, verified live |
| Verified | 29 September 2026, SGLang 0.5.20, A10G, Qwen3 8B, installed fresh as on the Install page: after SGLang's own cache was emptied, 1,792 of 1,824 prompt tokens came from Galahad; 27 September 2026, RTX A6000, Gemma 4 31B: 152,832 tokens served from Galahad |
| Needs | SGLang, the Galahad Python package, and a licence (Install) |
SGLang already reuses prompt memory very well while it is running. Its RadixAttention keeps the common beginnings of prompts in GPU memory and shares them across requests.
What it does not do is survive a restart. When the server stops, that memory is gone.
This sits underneath SGLang as the slower, durable layer it can fall back to. SGLang keeps doing what it is good at; this catches what it evicts, and keeps it across restarts.
⭐ It never replaces RadixAttention. It is the tier below it.
A coding assistant restarts after a deploy. SGLang's own cache starts empty. The first request for a file the team was working on all morning finds it here instead of recomputing it.
The full steps are in Install. In short:
export GALAHAD_CACHE_DIR=/var/galahad
export GALAHAD_KEY_FILE=/var/galahad/store.key
export GALAHAD_LICENCE_FILE=/etc/galahad/licence.lic
python -m sglang.launch_server --model-path <model> \
--page-size 64 \
--enable-hierarchical-cache \
--hicache-io-backend direct --hicache-mem-layout page_first_direct \
--hicache-storage-prefetch-policy wait_complete \
--hicache-storage-backend dynamic \
--hicache-storage-backend-extra-config \
'{"backend_name":"galahad","module_path":"galahad_vllm.sglang_backend","class_name":"GalahadHiCacheStorage"}'⚠ Keep --hicache-storage-prefetch-policy wait_complete. With SGLang's
default (timeout), SGLang can start computing before the memory has arrived
from disk, and then computes the whole prompt anyway.
⭐ No code, and nothing to register. SGLang's dynamic backend loads the
class named in the config. SGLang prints:
Creating dynamic storage backend 'galahad' (galahad_vllm.sglang_backend.GalahadHiCacheStorage)
⚠ In your own Python entry point you can also register it by name and use
--hicache-storage-backend galahad:
from galahad_vllm.sglang_backend import register_sglang_backend
register_sglang_backend()⭐ Memory survives a restart. What was saved before SGLang stopped is found again after it starts.
⭐ It fails open. If the store fails on every call, Galahad reports "not found", counts the error, and your server keeps serving.
SGLang's hierarchical cache uses one storage backend at a time. Galahad is one
you can choose, in place of another such as mooncake or file. Switching is a
change in the launch settings, not in your application.
⚠ The size of the win on a real SGLang workload is not published. If a figure matters to your decision, ask for it to be measured on your workload rather than inferred from the vLLM results.
⚠ Partial prefix hits are not offered on this path. It stores and returns whole entries. SGLang's own RadixAttention continues to do prefix work above it.
| Symptom | Cause | Fix |
|---|---|---|
SGLang does not list galahad
|
used --hicache-storage-backend galahad without registering |
use dynamic with the config above, or call register_sglang_backend() first |
no Creating dynamic storage backend 'galahad' line |
--enable-hierarchical-cache missing |
the disk tier sits under SGLang's hierarchical cache; turn it on |
| nothing found after a restart | store path changed |
GALAHAD_CACHE_DIR must point at the same store as before |
| errors counted, server still serving | fail open is working | fix the underlying store; nothing is being cached meanwhile |
| expecting partial prefix hits | this path stores and returns whole entries | use the vLLM Connector where prefix reuse matters |
- vLLM Connector: the same idea for vLLM
- Install · API Reference