Skip to content

2.2.28 Backend Halogen

Jump to bottom
av edited this page Sep 13, 2026 · 1 revision

Halogen Flash Server

Handle: halogen
API: http://localhost:34950/v1

Halogen serves Qwen3.8-Flash-Next through an OpenAI-compatible API on AMD Strix Halo (gfx1151). It is an API backend without a browser UI. It supports chat completions, streaming, tool calls and the Responses API.

Starting

Requires Linux, an AMD Strix Halo GPU with /dev/kfd and /dev/dri, and a 128 GB unified-memory machine. There is no CPU, NVIDIA or other AMD GPU fallback. Reserve at least 120 GiB of disk space for the model bundle, plus space for the container image. Upstream recommends a dedicated machine: other loaded models and large workloads can prevent startup or cause severe disk thrashing.

harbor pull --no-defaults halogen
harbor up --no-defaults halogen

First start downloads approximately 118 GiB into services/halogen/models. Downloads resume after interruption. Subsequent starts reuse the checkpoint, but a newer image may refresh its quality sidecar. Model loading takes several minutes after download. The healthcheck allows an hour for initial startup and probes the engine through the API.

Once healthy, list models and send a real request:

curl -fsS http://localhost:34950/v1/models
model=$(curl -fsS http://localhost:34950/v1/models | jq -r '.data[0].id')
jq -n --arg model "$model" \
  '{model: $model, messages: [{role: "user", content: "What is 2 + 2? Answer briefly."}], max_tokens: 256, temperature: 0}' \
  | curl -fsS http://localhost:34950/v1/chat/completions \
      -H 'Content-Type: application/json' --data-binary @-

Configuration

Environment Variables

Set these through harbor config, for example harbor config set halogen.ctx 32768:

Variable Default Purpose
HARBOR_HALOGEN_HOST_PORT 34950 Host API port
HARBOR_HALOGEN_IMAGE ghcr.io/peonist-ai/halogen-flash-server Upstream image repository
HARBOR_HALOGEN_VERSION 0.6.2 Image tag
HARBOR_HALOGEN_WORKSPACE ./services/halogen Workspace containing the models directory
HARBOR_HALOGEN_DOWNLOAD peonist-ai/halogen-qwen3.8-flash-next Hugging Face model bundle to download when absent
HARBOR_HALOGEN_CTX 262144 Maximum context per request
HARBOR_HALOGEN_KV_POOL_POSITIONS 524288 Shared KV pool capacity across requests
HARBOR_HALOGEN_KV_SLOTS 4 Concurrent request slots
HARBOR_HALOGEN_MAX_TOK 32768 Prefill chunk size, not the response token limit

After preparing a complete model bundle, disable outbound downloads with harbor config set halogen.download ''. Point the workspace at the parent of an existing flat models directory to reuse it. A Hugging Face cache snapshot alone may contain symlinks whose targets are outside the mount.

Use harbor env halogen for additional upstream variables. For example, harbor env halogen HALOGEN_VISION_TOWER 1 enables the vision sidecar included in the bundle. Contexts above 262144 require an explicit HALOGEN_ROPE_YARN factor and careful memory sizing. See upstream configuration before changing these settings.

Volumes

${HARBOR_HALOGEN_WORKSPACE}/models is mounted read-write at /models. It persists the checkpoint, tokenizer and sidecars across container removal. The upstream image controls ownership of downloaded files. Harbor does not remove them when stopping the service.

The container receives GPU devices, host IPC and unlimited locked memory as required by upstream. Only API port 8731 is published. The internal engine remains inside the container. The API has no authentication by default, so expose it only to trusted clients.

Integration with Harbor

harbor up --no-defaults webui halogen

Open WebUI waits for Halogen's healthcheck and receives http://halogen:8731/v1 in its OpenAI configuration. Select the model exposed by Halogen. Other clients on the Harbor network can use the same URL with a placeholder key such as sk-halogen. Host clients use http://localhost:34950/v1.

Troubleshooting

harbor logs halogen

Logs follow by default. For a finite diagnostic snapshot, use docker logs --tail 100 harbor.halogen (adjust the container prefix if configured).

  • Missing GPU devices: verify this is a supported Linux Strix Halo host with its AMD driver loaded.
  • Out of memory or very slow prompts: free other model workloads and reduce the KV pool or prefill size. Lowering context alone does not lower an explicitly configured pool.
  • Download failure: check available disk space and model-directory permissions, then restart to resume.
  • Unhealthy during a long first download: inspect progress in logs. Health becomes successful once download, model loading and the engine probe complete.

Links

Home | CLI Reference | Services | Adding New Service | Compatibility

Clone this wiki locally