0.1.45-poly.3

Flint 40B-A2.5B, Recursive's model for coding agents, faster than llama.cpp on the two machines it was tried on;
Gemma 4, Qwen3.6 and the other strata-poly models on AMD cards under Windows; and fixes to setup.

What changed since 0.1.45-poly.2

• Flint 40B-A2.5B by Recursive: ./setup.sh --family flint, or pick it in START-HERE.bat. Four sizes from
  Unsloth: UD-Q4_K_XL, UD-Q3_K_XL, UD-Q6_K_XL and Q8_0. Beside an 8 GB card setup takes UD-Q3_K_XL below about 40 GB
  of RAM: on a 32 GB laptop UD-Q4_K_XL's file did not stay in RAM. On an RTX 4070 laptop UD-Q3_K_XL reads a prompt
  of 29,000 tokens at 2,000 tokens/s and writes a short chat at 61 to 63; llama.cpp a37cc46104 read it at 390 and
  wrote after it at 35 there. On an RTX 5090 UD-Q4_K_XL reads it at 12,900 tokens/s and writes at 251; llama-bench
  reads 29,816 tokens at 12,470 and writes at 222. Asked for by hum-ma, HardeNNN and MaverickBlue
  (issue 16). Details: docs/FLINT.md.
• The product of a prompt batch with a router's 128 or 256 outputs goes in tiles, with the same bits as before:
  Flint's router takes 94 ms of a 29,816-token read on the 5090 instead of 241. Qwen3.6, Ornith, Tiel Coder,
  MiniCPM-V, DeepSeek V4 and Gemma 4 take the same kernel.
• Setup compiles strata-poly for an AMD card on Windows instead of refusing the models there: AMD's ROCm wheels, the
  Visual Studio C++ build tools and ROCm's compiler, with the ROCm libraries of the ready-made main engine. On an RX
  9070 XT with Tiel Coder UD-Q3_K_XL it read a 7,376-token prompt at 2,750 tokens/s and wrote at 55 to 75. That is
  the one card and model run so far; setup says so with a warning. By berkkarahan
  (PR 53).
• Your own GGUF: a whole file of the right architecture put in setup's own models folder now gets the way to run it
  (its own folder and --gguf-dir) instead of "delete it and run setup again". A file whose experts' gate and up
  have different types, which the engine cannot run, is refused before the install with the layer and the types,
  instead of stopping the engine at its first start (issue 54).
• --check-batch gives the attention caches their VRAM at load, so the batched phase no longer moves experts to the
  CPU and the comparison no longer reports differences that come from that. Found by
  donschen (issue 55).
• STRATA_GDN_MMA=1 reads the linear-attention layers of Qwen3.6, Ornith 1.5, Tiel Coder and MiniCPM-V with
  llama.cpp's tensor-core kernel. A 32K prompt read about 12% faster with Qwen3.6 and MiniCPM-V on an RTX 5090 and
  12 to 18% faster with Qwen3.6 and Tiel Coder on an RTX 4070 laptop; writing is unchanged. Off unless set; RTX 30
  cards and newer. By vivekpokale21 (PR 65).
• The chat page can regenerate an answer and step through its variants (arrows and an n/m counter, kept with the
  chat). By Ziomd (PR 74).
• --kv int8 and --kv int8k keep Gemma 4's attention caches in int8 (int8k: the keys only), opt-in; fp16
  stays the default. By smashdog (PR 59).
• GLM 5.3 Flash no longer fails at its first token on a Tesla V100, or on an RTX 20 card with a CUDA 12 build: one
  of its kernels asked for more registers than a block of 1,024 threads can have. Found by
  llollaalloll (issue 73).
• Setup goes on in a PolyStrata folder that was moved after it was installed: when the cmake in .venv still names
  the old place, setup installs it again instead of stopping at the engine build with "Fatal error in launcher"
  (issue 78). On AMD cards setup now runs ROCm's rocm_sdk through
  the venv's Python for the same reason: its rocm-sdk launcher named the old place too.
• Setup no longer stops with "unknown target 'strata-poly'" when it builds the engine again in a folder it built
  before and finds nvcc as nvcc.EXE where the folder has nvcc.exe: CMake took that for another compiler and
  dropped its settings.
• On Linux with an AMD card, the web page's GPU tiles of a strata-poly model said "not readable": setup now writes
  "backend": "hip" in those configs on Linux too, so run setup again to get it. Found by
  mkw432 (PR 71).
• A GLM reply now stops at <|user|> even when the GGUF names no end of turn (REAP50-Q3_K_M ran on to max_tokens),
  reported by whp199 (issue 70).
• Docs: the RX 6600 series runs setup's own engine without HSA_OVERRIDE_GFX_VERSION, and an override left from an
  older guide is what makes it fail with "device kernel image is invalid"
  (discussion 49); community results for Tiel Coder and
  MiniCPM-V on an RTX 4070 Ti SUPER, four models on an RX 6700 XT with a Ryzen 5 5600X (mkw432,
  PR 71), and Ornith 1.5 122B on an RX 7900 XTX with an R9700
  (Danielkgr).

The engine changed, so the zip below is a new ready-made Windows engine. It gives 0.1.45-poly.2's greedy tokens
with Qwen3.6, LFM2.5, Gemma 4, Ornith 1.5, Tiel Coder and MiniCPM-V on an RTX 4070 laptop (with and without the
MTP drafts), and with Qwen3.6, LFM2.5, MiniCPM-V, GLM 5.3 Flash and DeepSeek V4 Flash on an RTX 5090, and it
reads prompts and writes at least as fast as poly.2's engine there.

Install

Download the source below, or git clone the repository. Then run START-HERE.bat on Windows or ./setup.sh on
Linux. An earlier install updates with UPDATE.bat or ./update.sh.

Limits

• The nine added models are experimental.
• The ready-made engine is for Windows and CUDA 13 only; CUDA 12 and AMD cards compile on your PC.
• GLM 5.3 Flash and DeepSeek V4 Flash need 70 to 100 GB of RAM, and about 8 GB more than their files in a container.
  A DeepSeek file larger than the RAM runs several times slower than llama.cpp with the same file.
• On Windows, AMD cards have run one strata-poly model on one card (Tiel Coder on an RX 9070 XT); the others are
  untried there.

Most of this project is the work of Strata's author and contributors.

Being worked on now: reading experts from the SSD faster when a model's file does not fit in RAM, as with Flint's UD-Q6_K_XL and Q8_0 on a 32 GB laptop.