Skip to content

Uno for vLLM 0.1.0

Choose a tag to compare

View all tags
@mjbrn mjbrn released this 08 Sep 06:19
· 67 commits to main since this release
v0.1.0
5d9078c

Uno for vLLM 0.1.0

Run IFM's Uno diffusion adapter through vLLM's OpenAI-compatible server. This release brings native two-pass Uno decoding to Linux AMD64 and ARM64, with asynchronous scheduling, draft-only LoRA routing, prefix caching, private draft graph replay and seed-row reuse.

Get started

Docker selects the right image for your CPU architecture. Use a Linux host with an NVIDIA GPU, a compatible NVIDIA driver and the NVIDIA Container Toolkit.

docker pull ghcr.io/brntech/vllm-uno:0.1.0
docker run --rm --name vllm-uno --gpus all --ipc=host \
  -p 127.0.0.1:8000:8000 \
  -v vllm-uno-hf-cache:/root/.cache/huggingface \
  ghcr.io/brntech/vllm-uno:0.1.0 \
  Qwen/Qwen3-8B s-sahoo/uno-qwen3-8B -- \
  --attention-config '{"flash_attn_version":2}'

This starts Qwen3-8B BF16 with its original Uno adapter, K=8, on one NVIDIA GPU. The FlashAttention 2 setting above covers the tested Ampere and Blackwell GB10 configurations. First startup downloads the pinned model and adapter, then warms up the runtime; subsequent starts reuse the named download cache. The API is available at http://127.0.0.1:8000/v1 once /health is ready.

See the configuration guide for memory settings, model revisions and source builds.

Tested on hardware

Both native images passed on RTX 3090 (Ampere, AMD64) and GB10 (Blackwell, ARM64):

  • Model and adapter loading, health and model endpoints.
  • Greedy replies, sampling and streamed responses.
  • Shared-prefix requests and batches of 8 and 32 concurrent requests.
  • Active Uno drafting and acceptance, private graph replay and seed-row reuse.

Each image also passed 16 package and HTTP-gate tests, matched the source package and retained all 17 checked compiled libraries from its pinned upstream base. The shared source passed 368 focused CPU tests.

The attached validation record identifies the images, source, configuration and test results. The validation guide includes matched plain/Uno comparison procedures for other configurations. Performance measurements and methodology are in the paper and companion evidence.

Source and downloads

The source archive, validation record and SHA-256 checksums are attached. The same source tag supports local builds for either CPU architecture. BroadNet's code and release tools are licensed under Apache-2.0; model and adapter weights download from their original repositories.

  • Source tag: v0.1.0, commit 5d9078c83b6082be73cddc430b07fb516612076a.
  • Container: ghcr.io/brntech/vllm-uno:0.1.0 (also available as latest). No registry login is required.
  • Immutable image index: sha256:2780aa2d0b4c69e4ff8ac397b4c055aef5cf1a1464288d73a5a27bff10bb1224.
  • Source archive SHA-256: eb4a1550a1b95c8b83a44da5b85459cfa390a448c62f4bf1336e475f28223eb2.

Attribution

An independent community implementation by BroadNet Research Team. The Uno method and trained adapters are the work of IFM and the Uno authors.

Assets 5
Loading