Infernix windows-2026.10.11

Infernix windows-2026.10.11

AI disclaimer: Infernix was developed with AI (mostly Claude Opus 5.5). It is hobby work and
is likely to be neither complete nor entirely accurate.

Highlights

• Faster long prompts on Qwen3.8-Flash-Next. The prefix cache now snapshots a long prompt inside
  the layer walk instead of ending a walk at each snapshot, which made the walk stream the uncached
  experts across PCIe again. The call planner prices the walk's spans accordingly. Against the first
  release (windows-2026.10.09), time to first token is 18 % lower at 128K tokens (14.7 s against
  18.0 s) and 12 % lower at 250K (30.1 s against 34.1 s).
• Faster Qwen3.8-Flash-Next decode: against the first release, 1-8 % faster on single requests
  from 8K to 250K tokens of context, and 3 % (one request) to 8 % (two requests) faster in the
  agentic replay.
  • Calls of up to eight columns, and every narrow W4A4 call, run on the one-column expert kernels
    (5- and 6-column MTP verification rounds 9-17 % shorter); W4A16 (Uncensored) calls of up to four
    columns too.
  • The one-column expert products convert FP4 weights to half2 in one instruction and multiply with
    HFMA2, and remain exact (integer row sums).
  • The expert down pass, on both the NVFP4 and the W4A16 (Uncensored) routes, is launched as a
    PDL dependent of gate/up.
  • The fused hyper-connection mixer is chained with PDL.
  • The MTP drafter's expert kernel no longer has shared-memory bank conflicts.
• Faster Qwen3.8-Flash-Next loading: the 63 GiB pinned expert block is zeroed while it is read
  and page-locked once at the end. Dense8 loads in 13.7 s instead of 19.7 s.
• Better use of the CPU and the expert cache on Qwen3.8-Flash-Next:
  • At startup the engine times its own CPU expert workers, and plans both the decode miss split and
    the prefill CPU split from the measured rates rather than constants fitted to an i9-13900K. With
    fewer CPU workers this is up to 12 % faster in decode and 11-19 % faster in short-prompt prefill.
  • Expert-cache counts are now halved every 8,192 layer calls, so the cache follows a change of topic
    (9-12 % fewer misses on mixed requests).
  • Lending cache frames to a prompt now evicts the lowest-scoring experts rather than a near-random
    slice of the cache, and moves the displaced experts in one kernel launch.
  • On Windows, the CPU miss service no longer oversleeps a 15.6 ms timer tick after an idle gap.
  • Direct expert reads from the artifact keep sixteen 8 MiB reads in flight (7.0 GB/s, up from
    5.4-5.9 GB/s on a 990 PRO).
  • Two wide-scoring QSA prompt-attention blocks fit per SM (BF16 key staging, bitwise-identical
    scores).
• Prefix cache:
  • On Qwen3.8-Flash-Next a request's own earlier snapshots (its system-block boundary and history
    ladder) stay retained instead of being evicted first, so edits of the history and side calls
    over it resume there. In the agentic replay 84.7 % of prompt tokens come from cache, as in the
    first release.
  • Endpoints a conversation has gone past without resuming from them (clients that re-render the
    reply) are superseded, so they no longer hold snapshot slots for good.
• Fixes from GitHub issues and pull requests:
  • #18: on Qwen3.8-Flash-Next, a ~480K-token
    resume after the 60 s idle shrink failed with HTTP 503 ("more than an idle Engine has"), because
    one KV growth could take at most a quarter of the expert frames. One growth now takes as many as
    it needs.
  • #17: a server started over OpenSSH or as a
    service (Session 0) spun in DXGI display enumeration before loading weights.
  • On Qwen3.8-Flash-Next the /metrics speculative-decoding counters
    (infernix_spec_decode_*_total) stayed at 0; they now count as on Qwen3.5. Thanks to @mdbro.
  • Single-config builds (Ninja) staged the FFmpeg/curl/zlib DLLs into a directory nothing ran from,
    so infernix-serve.exe failed at startup with a missing DLL. Thanks to @mdbro, who also
    contributed three build cleanups in the Ops code.
• Linux fixes (cherry-picked from the community fixes branch):
  • #7: Flash-Next decode on Linux ran at about
    20 tok/s, because every n-gram read created and destroyed a kernel AIO context. Contexts are now
    kept.
  • #3: cicc in CUDA 13.2 and 13.3 on Linux
    crashed compiling the Q8 GDN input GEMM. Thanks to @kido5217 for the report and the fix.
  • #5: with SSD-only experts, a long prompt's
    layer walk next to another request's decode failed with "the host expert service stopped
    answering".

Benchmarks

The README's benchmark tables are re-measured on this release, against Strata 0.1.42 and NInfer
81c8ce09 (with the Windows port), and now include the first Infernix release (windows-2026.10.09).
Every timing comes from the client's clock for every engine. The previous agentic tables mixed in the
engines' own timers, and NInfer's prefill timer stopped about 8 ms before the work ended.

• Qwen3.8-Flash-Next against Strata: 5.4-7.0× sooner to the first token and 1.4-1.6× the decode
  speed from 8K to 250K tokens. In the agentic replay (Strata with two slots and batched MTP), 5.2 s
  to the first token on average against 19.3 s, 84.7 % of prompt tokens from cache against 72.1 %,
  and the workload done in 7.2 minutes against 20.1. With that configuration Strata failed 5-7 of
  68 requests per seed.
• Qwen3.8-27B against NInfer: long prompts prefill 1.36× (128K) and 1.53× (230K) faster; in the
  agentic replay (five seeds, 150K context, the largest at which NInfer starts with two requests and
  DFlash2) the first token comes 33 % sooner on average and the median falls by 59 %, with 85.4 % of
  prompt tokens from cache against 79.2 %.
• Against the first release on the 27B the single-request results are identical. In the agentic
  replay the engine does the same work at the same speed (prompt time per request equal, lane time
  2,253 s against 2,277 s, one-request decode +1 %), but the average time to the first token is
  12 % higher (2.8 s against 2.5 s; median +5 %, 90th percentile +4 %): requests waited longer for
  one of the two lanes, on four of the five seeds, behind sampled replies that ran longer.

Behaviour notes

• No breaking changes: the flags, the APIs, the request-log schema (27) and the artifact formats are
  unchanged.
• The -1.8 % prefill gap listed for windows-2026.10.09.2 was not a slowdown. The old build's prefill
  timer stopped about 8 ms early, and by wall time per repetition the hybrid-cache builds are
  0.4-0.5 % faster.

Downloads

• infernix-2026.10.11-windows-x64.zip: infernix-serve.exe with its FFmpeg, curl and zlib
  DLLs, the licence of every component it contains or loads (licenses/) and
  THIRD_PARTY_NOTICES.md.
• ffmpeg-9.0.1-source.zip: not needed to run Infernix. It is provided for licence
  compliance: the FFmpeg DLLs in the release zip are LGPL-2.1-or-later, and this is their
  corresponding source.

Requirements: an RTX 50-series GPU (sm_120a), an NVIDIA driver of 580 or later (CUDA 13), and the
Visual C++ Redistributable for Visual Studio 2015-2026. No CUDA toolkit is needed. Linux users build
from source (see the README).

Verification and known gaps

Verified on the tagged build (RTX 5090 at PCIe Gen5 x8, Windows 11, VS 2026 / CUDA 13.4):

• The unit test suite (212 tests, 9 of them skipped because they need a model artifact) passes.
• Qwen3.8-27B real tests on the official NVFP4 artifact: hybrid prefix cache, readout, preemption
  (no drafter and MTP), tools (graph, replay, cancel, MTP), grammar, constraint, scoring, prefix,
  native transactions, DFlash2, n-gram lifecycle and n-gram concurrency pass.
• Qwen3.8-Flash-Next real tests on the Dense8 artifact: prefix cache, preemption, /v1/decide,
  expert-read fault and VRAM pressure pass.
• The release zip, unpacked with only System32 on PATH, serves Dense8: models, chat, streaming, a
  tool call, Anthropic Messages and Responses.
• The pull requests' own checks: a clean Windows build and a WSL build (GCC 13.3, CUDA 13.4) with no
  nvcc warnings; Flash-Next /metrics speculative counters equal to the response timings; a
  ~199K-token cold prompt that returned HTTP 503 before #18 now completes.

Known gaps:

• The 27B preemption test first failed in the release gate: it read the engine's published counters
  before the worker had published the request's completion. The test is fixed (it waits for the
  worker boundary again, as before windows-2026.10.09.2); it then passed in the gate and in two more
  runs in the gate's order. The engine was not changed.
• The DFlash2 and n-gram real tests now run on the official artifact; the Q6 artifact they used is
  no longer kept.
• Four n-gram real tests (graph planning, archive, thinking, stop in chat) were not run for this
  release.
• The #17 fix (startup in Session 0, over OpenSSH or as a service) was not exercised in Session 0.
• The README's benchmarks were measured on 23d310d6, before the pull requests; their changes
  (KV growth, metrics, build fixes) were not re-benchmarked.
• Linux: built and unit-tested (CPU subset) under WSL; the real tests were not run on Linux.

Support

If you would like to help pay for the Claude Code subscription Infernix is developed with, you can
sponsor me on GitHub. Please also consider supporting
Neroued, who created NInfer.

Infernix windows-2026.10.09.2

Infernix windows-2026.10.09.2

AI disclaimer: Infernix was developed with AI (mostly Claude Opus 5.5). It is hobby work and
is likely to be neither complete nor entirely accurate.

Highlights

• Qwen3.8-Flash-Next Uncensored: runs
  Qwen3.8-Flash-Next-Uncensored-NVFP4-Infernix,
  a conversion of
  orcarouter/Qwen3.8-Flash-Next-Uncensored-NVFP4.
  Its routed experts are weight-only 4-bit (W4A16), served by new exact expert routes on the CPU and
  on the GPU (integer tensor cores for prefill); the dense layers stay bit-exact FP8. It needs the
  52 GB PLE n-gram volume from the
  NVIDIA conversion's repository
  (the same file: the abliteration left the n-gram table unchanged).
• One prefix cache: every model now runs on the hybrid prefix cache (content-addressed 64-token
  KV blocks and sparse state snapshots, on the GPU and in one pinned Host pool). The original
  continuation/checkpoint cache and its code are removed.
• Fix for #1: a request that recomputed
  already-cached blocks pinned two pages per block, so with --kv-capacity equal to --max-context
  the pool filled and the server answered 503 to every later request. A recomputed block now takes
  over its unused cached copy. The capacity error that remains possible now fails only the requests
  holding lanes on Qwen3.8-Flash-Next, and the server keeps serving.

Breaking changes

• --use-original-prefix-caching and --device-state-slots are removed. --device-snapshot-slots
  sizes the Device snapshot slots; --host-context-mib sizes the prefix cache's Host pool and is
  rejected with --no-prefix-reuse.
• The JSONL request log is schema 27. context_cache reports selections (root, prefix), state
  operations, partial_tail_cow_pages and Device occupancy. The Host-context, transfer, capture
  and spill fields, revoked_checkpoints, the snapshot recovery route and the checkpoint
  reuse path are gone.
• Prometheus: infernix_checkpoint_selections_total is now infernix_prefix_selections_total;
  the infernix_host_context_* gauges and infernix_context_transfer_* counters are removed.
• OpenAI Responses no longer derives an Engine session key from response IDs: reuse is by content.

Behaviour notes

• An exact repeat of a very short prompt (under one 64-token block) is no longer served from the
  cache, and after a generation shorter than 64 tokens (a short tool call, say) the next turn
  resumes from the turn's generation opener rather than the end of the reply. The prefix cache
  keeps a snapshot only a block or more past the previous one; infernix-serve already used this
  cache by default, so served behaviour is unchanged.

Downloads

• infernix-2026.10.09.2-windows-x64.zip: infernix-serve.exe with its FFmpeg, curl and zlib
  DLLs, the licence of every component it contains or loads (licenses/) and
  THIRD_PARTY_NOTICES.md.
• ffmpeg-9.0.1-source.zip: not needed to run Infernix. It is provided for licence
  compliance: the FFmpeg DLLs in the release zip are LGPL-2.1-or-later, and this is their
  corresponding source.

Requirements: an RTX 50-series GPU (sm_120a), an NVIDIA driver of 580 or later (CUDA 13), and the
Visual C++ Redistributable for Visual Studio 2015-2026. No CUDA toolkit is needed. Linux users build
from source (see the README).

Verification and known gaps

Tested on one RTX 5090 (PCIe Gen5 x8), Windows 11, CUDA 13.4, on this release's build:

• All 211 unit tests pass.
• Qwen3.8-27B real tests pass (NInfer's official NVFP4 artifact; a Q6 DFlash2 artifact for the
  DFlash2 and n-gram tests): hybrid prefix cache, the prefix-cache scenarios, preemption (with and
  without MTP), tool calling (graph, replay, cancel, MTP), grammar and constraints, readout, causal
  scoring, native transactions, DFlash2 and the n-gram lifecycle.
• Qwen3.8-Flash-Next real tests pass on the Uncensored artifact (prefix cache, preemption, decide,
  expert-read fault recovery, VRAM pressure) and on Dense8 (prefix cache, decide, VRAM pressure).
• The release zip, unpacked with only System32 on PATH, serves both the Dense8 and the
  Uncensored Flash-Next artifacts: model list, chat, streaming, a tool call, Anthropic Messages and
  Responses.

Known gaps:

• Prefill speed (Qwen3.8-27B, prefix cache disabled): against the build before the cache change,
  1,024-token prompts measured −0.5 %, 4,096-token prompts −1.8 % and 16,384-token prompts −0.2 %
  (infernix_bench, two runs per arm in A/B/B/A order, six repetitions each). A profile shows the
  same kernels, launch counts and API calls on both builds; the difference sits in the duration of
  unchanged GEMM kernels, and its cause is not identified. Decode measured −0.7 % (27B, DFlash2) to
  +1.8 % (Flash-Next).
• N-gram concurrency real test: ngram_concurrent fails on this release (a copy-drafting lane
  ends its turn after 40 tokens). It also fails on the previous build, at a later check, so it is
  not a regression from this release; it is a test still to be reworked.
• Device-only prefix cache: with no Host pool, a request that branches from a cached prefix and
  continues past it can replace the snapshot the parent conversation would resume from, so the
  parent's next turn recomputes from an earlier point. Outputs are unaffected; only reuse is lost.

Support

If you would like to help pay for the Claude Code subscription Infernix is developed with, you can
sponsor me on GitHub. Please also consider supporting
Neroued, who created NInfer.

Infernix windows-2026.10.09

Infernix windows-2026.10.09

The first release of Infernix as its own project: a C++/CUDA inference engine for Qwen models on a
single NVIDIA RTX 50-series GPU, grown from NInfer by
Neroued.

AI disclaimer: Infernix was developed with AI (mostly Claude Opus 5.5). It is hobby work and
is likely to be neither complete nor entirely accurate.

Highlights

• Qwen3.8-Flash-Next on one RTX 5090: experts in pinned host RAM with a VRAM expert cache, CPU
  computation of cache misses and the n-gram table on NVMe. In a replayed agentic coding workload
  against Strata 0.1.41 on the same machine: 5.9 s average time to first token against 32.9 s,
  84.6 % of prompt tokens from cache against 53.8 %, 125 against 86 tok/s decoding one request, and
  7.8 against 22.3 minutes of wall time.
• Qwen3.8-27B: a hybrid prefix cache built for Qwen's linear-attention layers (90 % of prompt
  tokens from cache in the agentic replay, half NInfer's average time to the first token), faster
  prefill and DFlash2 + n-gram drafting, at NInfer's quality (perplexity 4.468 against 4.479).
• Built for agents: OpenAI Chat Completions and Responses and Anthropic Messages, robust tool
  calling, constrained decoding, thinking control, prefix cache persistence (also every N minutes
  with --prefix-cache-save-mins), and recovery from a failed expert read without failing the
  server.

See the README for the benchmarks and the
Qwen3.8-Flash-Next guide.

Downloads

• infernix-2026.10.09-windows-x64.zip: infernix-serve.exe with its FFmpeg, curl and zlib DLLs,
  the licence of every component it contains or loads (licenses/) and
  THIRD_PARTY_NOTICES.md.
• ffmpeg-9.0.1-source.zip: not needed to run Infernix. It is provided for licence compliance:
  the FFmpeg DLLs in the release zip are LGPL-2.1-or-later, and this is their corresponding source
  (the FFmpeg source release and the vcpkg port that built them).

Requirements: an RTX 50-series GPU (sm_120a), an NVIDIA driver of 580 or later (CUDA 13), and the
Visual C++ Redistributable for Visual Studio 2015-2026. No CUDA toolkit is needed. Linux users build
from source (see the README).

Models

• Qwen3.8-Flash-Next-NVIDIA-NVFP4-Dense8-Infernix (recommended)
• Qwen3.8-Flash-Next-NVIDIA-NVFP4-Infernix (bit-exact)
• Qwen3.8-27B-NVIDIA-NVFP4-NInferV3
• Qwen3.8-27B-Quasar-NVFP4-NInferV3
• Qwen3.8-27B-Uncensored-NVFP4-NInferV3

Verification and known gaps

• Windows: unit tests of the changed code and the Qwen3.8-Flash-Next real tests (prefix cache with
  the periodic save, preemption, /v1/decide, expert-fault recovery, VRAM pressure) pass.
• Linux (WSL2, Ubuntu 24.04, CUDA 13.4): the full build and 210 of 211 unit tests pass (the other
  needs PyTorch); the Qwen3.8-Flash-Next prefix-cache real test passes and the CLI's greedy token
  ids equal Windows'. The remaining Linux real tests and the serve smoke test were not run for this
  release.
• The Qwen3.8-Flash-Next vision route has no FP64 reference test yet.
• Measured with the GPU in a PCIe Gen5 x8 link; x16 results are welcome.

Support

If you would like to help pay for the Claude Code subscription Infernix is developed with, you can
sponsor me on GitHub. Please also consider supporting
Neroued, who created NInfer.