0.1.45-poly.7

The strata-poly models keep less of your RAM at the same speed, a new opt-in setting about halves it again at some cost in speed, and a client that saves a session no longer ends the engine.

What changed since 0.1.45-poly.6

• The eleven models on strata-poly keep less of your RAM, with no setting. The engine no longer holds the parts of
  the model file that are already on the graphics card, and it drops cuBLAS's pages between requests. On an RTX 4070
  laptop (8 GB), Qwen3.6-35B-A3B UD-Q3_K_M after a 32,000-token prompt held 17.9 GiB with 0.1.45-poly.6 and holds
  13.1 GiB now, and writes as fast; under Linux it went from 17.8 to 12.6 GiB. Pinned memory went from 1.2 to 1.4 GiB
  down to 0.17. Reading a prompt needs 350 to 400 MiB less VRAM, which the expert cache takes (Qwen3.6 keeps 2,712
  experts in VRAM where 2,444).
• A parked conversation goes to a temporary file between requests instead of staying in the engine's memory: with
  a 32K conversation parked the engine held 12.58 GiB where 13.69 on the laptop, and the conversation came back in
  163 ms where 121. The image encoder on the CPU starts with the first picture and stops after 120 s without one,
  where it held 1.1 GiB from the server's start; the first picture after a pause waits 0.9 to 1.9 s for it.
• New and off by default: --ram-experts-mib 6144 holds 6 GiB of RAM for the experts outside VRAM, while it
  answers too, and leaves the others in the system's file cache, where other programs can take that memory. It
  writes slower, and we are still working on that. On the laptop with Qwen3.6 UD-Q3_K_M it held 6.5 to 7.3 GiB where
  11.7 to 14.3. Over three runs it wrote within 3% of the default in the first short chat, 3 to 8% slower after a
  32K prompt and 8 to 13% slower in a short chat after that. Under Linux (an RTX PRO 6000 held to about 8 GB) it
  held 7.0 to 7.7 GiB where 12.3 to 14.3 and wrote 7 to 14% slower. To use it, add "--ram-experts-mib", "6144" to
  the "args" list of strata-<model>.json and restart. The numbers are in the
  README, and how it works in
  docs/BACKENDS.md.
• The server answers a session save or restore (POST /slots/0?action=save with --slot-save-path) with 501 on
  the strata-poly models, which have no disk sessions, instead of ending the engine. A proxy that saved sessions
  restarted Tiel Coder over and over. Reported by maverde73
  (issue 114).
• STRATA_POLY_STREAM_STAGE=1, off unless set, copies the experts into the staging buffer with streaming stores.
  By vivekpokale21 (PR 109).
  The answers are the same with it and without it; on the RTX 4070 laptop it read and wrote as fast.
• New community results in
  docs/COMMUNITY_BENCHMARKS.md:
  an RX 7900 XTX with Clover Q4_K_M by brocasm
  (issue 115), a GTX 1650 4 GB with Clover and MiniCPM-V by
  LionelColaso (PR 111) and an
  RTX 3080 with Tiel Coder by HexhamAllstar
  (PR 113).

The engine changed, so the zip below is a new ready-made Windows engine. With the expert cache at 0.1.45-poly.6's
size (--stage-mib 2048 --stage-read-mib 0) it gives 0.1.45-poly.6's greedy tokens on an RTX 4070 laptop under
Windows with Qwen3.6, LFM2.5, Gemma 4, Ornith, Tiel Coder, MiniCPM-V and Clover, with STRATA_FATTN=0 STRATA_FDEC=0
and, for the Qwen3.6 family, with the defaults too, and with Qwen3.6 on an RTX PRO 6000 under Linux. Flint gave
them for a short prompt; its 3,000-token prompt was not run, since poly.6's engine holds its whole 18.5 GB file and
the laptop did not have the RAM free. With the defaults the cache holds more experts, the CPU and the card share the
work another way, and an answer can part from poly.6's where two tokens are close. With the experts held still,
--ram-experts-mib 6144 gives the same tokens as the default: Qwen3.6, LFM2.5, Gemma 4, Flint and Clover on the
laptop, Qwen3.6 under Linux. An install of 0.1.45-poly.6 updated to it with UPDATE.bat and kept its config as it
was. GLM 5.3 Flash, DeepSeek V4 Flash and Kolibri 1 were not run for this release.

Install

Download the source below, or git clone the repository. Then run START-HERE.bat on Windows or ./setup.sh on
Linux. An earlier install updates with UPDATE.bat or ./update.sh.

Limits

• The eleven added models are experimental.
• The ready-made engine is for Windows and CUDA 13 only; CUDA 12 and AMD cards compile on your PC.
• GLM 5.3 Flash and DeepSeek V4 Flash need 70 to 100 GB of RAM, and about 8 GB more than their files in a container.
  A DeepSeek file larger than the RAM runs several times slower than llama.cpp with the same file.
• GLM 5.3 Flash, DeepSeek V4 Flash, Flint, Kolibri 1 and Clover have not been tried on an AMD card. On Windows, AMD
  cards have run one strata-poly model (Tiel Coder on an RX 9070 XT).
• --ram-experts-mib was measured with 6 to 12 GiB of experts outside VRAM, not with more.

Most of this project is the work of Strata's author and contributors.

Planned: SSD optimization for strata-poly, and a faster --ram-experts-mib.