Skip to content

v0.10.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 09 Nov 00:33
· 258 commits to main since this release
v0.10.0
97a43ec

ignis v0.10.0

RTX 30 and RTX 40 cards

  • The release binaries run on RTX 30 and RTX 40 cards (#474, #482, What it runs on): the Windows zip, the Linux tarball and the container image carry kernels for compute capabilities 8.6 and 12.0, and the release check refuses a binary that misses either.
    • RTX 3090 and RTX 3090 Ti: supported, on the 27B's 16 GB entries (qwen3.8-27b, its uncensored variant, Swift 1.5, #529).
    • RTX 4090 and the 16 GB RTX 40 cards (RTX 4060 Ti 16 GB, RTX 4080): verified for correctness on the same entries, by the release's gate on an Ada Lovelace card (#532). No speed is published for RTX 30 or RTX 40 yet: the figures will come from your reports.
    • The NVFP4 27B, Flash-Next and the image model need Blackwell. On an RTX 30 or 40 card the 27B's ids are served by their 16 GB entries, and a Blackwell-only entry is refused 409 model_above_gpu_generation before anything unloads (#481, ADR 0079).
  • A 24 GB card class, rtx-24gb (#480): --profile auto takes it on an RTX 3090 or 4090. It gives the 27B's 16 GB entry 1,048,576 tokens of context on eight lanes, DFlash2 with 7 drafts and vision with its tower in host RAM.
  • rtx-16gb is the 16 GB profile's name (#476); rtx50-16gb stays an alias, so existing commands keep working.
  • A prefill built for RTX 30 and RTX 40 (#503, ADR 0080): each trellis matrix is decoded once per chunk into INT8 and multiplied on the integer tensor cores, the prompt attention runs an 8-bit band of its own, and the gated delta net prefill runs ignis's own kernels, all in the forms these cards run at full rate. It is their default; --model-prefill-linear and --model-prefill-attention name the other routes, and GET /v1/config reports the ones in force (#505, #477).
  • The release gate on an 8.x card (#484, releases): the model's quality against its BF16 checkpoint, the kernels' tests and a server smoke test. It passed on an sm_86 card on 6234e00 and, for this release, on main adb5ed2, the last code commit before 0.10.0 (the release commit adds the version and the guide): top-64 KLD 0.03944 (limit 0.04081), top-1 93.33% (limit 92.85%), 127 of 127 kernel tests, a chat and two images answered. The sm_86 card was no longer available, so by the owner's decision this run went on an Ada Lovelace card (sm_89), which runs the same 8.6 kernels; its figures equal the sm_86 card's in every digit (finding).

A faster first token, on every card

  • The first token leaves with the prefill (#502, #518-#521): it is sent as soon as the prompt's last chunk is done, and the work that used to sit in front of it (the drafter's captures, the head on intermediate chunks) is off its path.
  • A wider prefill chunk while no lane is decoding (#489): --model-prefill-chunk-solo, auto by default (the solo width).
  • On Blackwell the 27B's gated delta net prefill runs ignis's own kernels (#531, #536, ADR 0085): 2.4-2.6x faster than the vendored ones, a time to first token 3-4% shorter at 8K and 64K on the 16 GB entry and 4.2-5.7% on the NVFP4 one, with the KLD against BF16 unchanged. A greedy answer can now differ from an earlier release's at a near-tie, where two tokens are almost equally likely.
  • On the RTX 5090 held to 16 GiB, the rtx-16gb load, the time to first token measured about 5% shorter at 8K and 3-6% at 64K than on 0.9.6.

small-card: a long-reply cell, and how its readings move

  • ignis-bench small-card measures decode over long replies (#541, #542, Measure a 16 GB card): a decode.long cell, coding prompts whose answers run to 460-760 tokens, is the figure to quote for a card. Each cell also records its own DFlash2 acceptance.
  • The short-reply readings drop with this release, and the card is not slower (#543). Since the first token leaves with the prefill, small-card times every round whose tokens it counts. Its coding cell's prompts end after 17 to 111 tokens, so its median read high on 0.9.x. An A/B on the RTX 5090 held to 16 GiB, 0.9.6 against this release with the same small-card, two clean pairs of alternated runs (a third pair ran beside a build and reads lower on both sides):
    • coding median 452-460 → 343-344 tok/s (−25%), pooled 401-403 → 367-369 (−8%);
    • long replies 457-466 → 448-458 tok/s (−1 to −2%, within the runs' spread);
    • agent turns 413-426 → 352-369 tok/s, most of it a different greedy text at a near-tie from the new GDN prefill: with the earlier kernels named, this release drafts those turns exactly as 0.9.6 did.
    • Records on #360 taken with a 0.9 release compare with each other as before, but not with a 0.10.0 coding reading: compare long replies.
    • The A/B's records, both arms, are attached below: small-card-ab-0.9.6-vs-0.10.0.zip.

Other changes

  • Build and CI: a cargo build no longer compiles the kernel's test executables (#460); the Linux build states CMake 3.28 and finds cudart beyond $CUDA_HOME/lib64 (#498); the workspace compiles on macOS again (#496); CI runs the builds beside sharded tests and caches the kernel build, about 12 minutes from 46-64 (#552); a fork's push publishes no development build (#495).
  • Fixes: VISION=0 turns vision off over a profile that loads it (#491); a start refused for a missing sidecar says how to get it (#497); two flaky Linux tests and the kv-disk tests' waits (#492, #554).
  • The guide: one table of which built-in model loads on which card (#528), a "Working in a fork" page (#500), and the records cleaned up to say what holds today (#544).

Decisions

  • ADR 0079: a catalog entry states the oldest GPU generation it loads on
  • ADR 0080: on sm_8x the trellis prefill decodes each matrix once per chunk and multiplies on the integer MMA
  • ADR 0081: an ADR's status is in its frontmatter, and a decision that changes is recorded by a new ADR
  • ADR 0082: the gate is KLD against BF16, DFlash2 on against off and the loop rate (supersedes ADRs 0014, 0015, 0021)
  • ADR 0083: the north-star is the best local engine for coding, for agentic work and as a companion
  • ADR 0084: ignis stands on its own
  • ADR 0085: on Blackwell too, the 27B's GDN prefill runs ignis's own kernels
  • ADR 0086: the builds run beside the tests, and nothing is published before they pass
  • Amended: ADRs 0016, 0017, 0018, 0023, 0024, 0026, 0029, 0030, 0032, 0045, 0052, 0063, 0064, 0070, 0077; accepted: ADRs 0042, 0065

Container image (linux/amd64)

podman run --rm --device nvidia.com/gpu=all -p 8000:8000 \
  -v /path/to/models:/models:ro \
  -e IGNIS_MODEL_ARTIFACT=/models/<name>.ninfer \
  ghcr.io/gpillon/ignis:0.10.0

(docker: --gpus all in place of --device.) The image carries
the CUDA runtime but no driver — the host's NVIDIA driver is
injected by the container runtime. It is built from the same
compile as the Linux tarball below, so the two always carry the
same binaries.

Binaries

The kernel is compiled for compute capability 8.6 and 12.0
(86;120a), so the same binaries run on:

  • RTX 30 cards (Ampere): the RTX 3090 and 3090 Ti are supported,
    on the 27B's 16 GB entries;
  • RTX 40 cards (Ada), on the 8.6 code: the RTX 4090 and the 16 GB
    cards are verified for correctness, and reports are welcome;
  • RTX 50 and RTX PRO 6000 cards (Blackwell).

The NVFP4 27B, Flash-Next and the image model load on Blackwell
alone. The engine needs a .ninfer artifact, which is not part of
this release. The Linux tarball needs the CUDA 13
runtime on the host; the Windows zip carries cudart64_*.dll.
Both need an NVIDIA driver new enough for CUDA 13.

Apache-2.0 (LICENSE, NOTICE); the vendored reference ops keep
their own terms (NOTICE-kernel).

Full Changelog: v0.9.6...v0.10.0

Assets 7
Loading