R9V v0.4.4 — CED on shares VRAM with the vision encoder

R9V v0.4.4 gives the uncensored profile's default --ced on mode about 0.4 GiB more free VRAM on each GPU, with the same speed and outputs.

What's new

• --ced on now shares VRAM with the vision encoder, the way --ced quality does since v0.4.3. Image prompts never use CED, so the vision encoder's weights (0.42 GiB per GPU) and the CED projector (1.76 GiB per GPU) no longer sit in VRAM side by side. They take turns in one VRAM region per GPU and are copied back in from RAM when the other one is needed. A swap takes 35–39 ms (projector) or 8–9 ms (vision) on GPU 0 and 262 ms or 63 ms on GPU 1, and only happens when a long CED prompt follows an image or the other way round.
• Where the VRAM was: besides the resident vision weights, on's CED chunks peak while running layers 0–15 uncompiled (0.84 GiB above the step's start, against 0.62 GiB for a normal chunk). The projector math itself adds nothing on top, so there were no buffers to trim as in v0.4.3's quality fix.
• RAM: on now needs 4.50 GiB more available RAM at start (60.8 GiB instead of 56.3), the same as quality, for the pinned copies. The doctor checks it. --ced off still needs 56.3 GiB and launches exactly as in v0.4.3.

Tested on the reference host

Clean install (fresh clone, fresh state and caches), first start in each mode, desktop apps holding about 1.2 GiB of GPU 0. Headroom at max context: the minimum free VRAM during first-start qualification, which includes a 130,941-token prompt at the 131,072 limit (target 1.5 GiB):

GPU 0 / GPU 1	v0.4.4	v0.4.3
on (default)	2.07 / 2.52 GiB	1.65 / 2.12 GiB
quality	2.04 / 2.49 GiB	2.04 / 2.49 GiB
off	3.67 / 4.15 GiB	3.68 / 4.15 GiB

• Prefill speedup with on: 1.59× at 12.8K tokens, 1.77× at 32K (v0.4.3: 1.58× / 1.76×).
• Warm decode: 39.1–39.2 ms/step median, against 40.9 for v0.4.3's on in the same session.
• CED's output is unchanged: a development build checked on the GPU that the projector in the shared region matched the file after every swap and gave bitwise the same results as a separately allocated copy (v0.4.3's layout). The vision encoder's output was bitwise identical to v0.4.3's for 9 images, before and after swaps.
• Exact requests after a CED request were bitwise identical to fresh exact runs; repeated greedy runs were identical; nothing recompiled after the first CED request.
• 24 alternating long-CED and image requests: VRAM stayed flat and every image was read correctly. A long CED request with three images sent during it: all completed.
• quality and off qualified in the same session; an unchanged restart reused its qualification (ready in 219 s).

Upgrading from v0.4.3

• The runtime descriptor changed, so an existing install qualifies again once on its next start (a few minutes), in any CED mode.
• No new downloads. Make sure 60.8 GiB of RAM is available at start with CED on (the doctor tells you).

Details: PR #43 and the changelog.

R9V v0.4.3 — CED quality fits next to the vision encoder

R9V v0.4.3 makes the uncensored profile's --ced quality mode fit on the reference host, with image support kept.

What's new

• --ced quality now passes first-start qualification on the dual-R9700 reference host. In v0.4.2 it failed there: GPU 0 dropped to 0.90 GiB free against the 1.5 GiB target.
• How: image prompts never use CED, so the vision encoder's weights (0.42 GiB per GPU) and the quality projector (1.79 GiB per GPU) no longer sit in VRAM side by side. They take turns in one VRAM region per GPU and are copied back in from RAM when the other one is needed. A swap takes 36 ms (projector) or 9 ms (vision) on GPU 0 and 266 ms or 63 ms on GPU 1, and only happens when a long CED prompt follows an image or the other way round. The projector's work buffers are also smaller.
• RAM: quality now needs 4.50 GiB more available RAM at start (60.8 GiB instead of 56.3) for those copies. The doctor checks it.
• --ced on (the default) and --ced off are unchanged.

Tested on the reference host

Clean install (fresh clone, fresh state and caches), first start with each mode, desktop apps holding 1.23 GiB of GPU 0:

	quality	on	off
Min free VRAM during qualification, GPU 0 / GPU 1	2.04 / 2.49 GiB	1.65 / 2.12 GiB	3.68 / 4.15 GiB
Prefill speedup at ~12.8K / ~32K tokens	1.47× / 1.62×	1.58× / 1.76×
Warm decode (ms/step)	35.8–41.6	36.0–43.2

• 24 alternating long-CED and image requests: VRAM stayed flat and every image was read correctly. A long CED request with three images sent during it: all completed.
• The vision encoder's output was bitwise identical to --ced on's for 8 test images, before and after swaps.
• Exact requests after a CED request were bitwise identical to fresh exact runs; repeated greedy runs were identical; nothing recompiled after the first CED request; an unchanged restart reused its qualification.

Upgrading from v0.4.2

• The runtime descriptor changed, so an existing install qualifies again once on its next start (a few minutes), in any CED mode.
• No new downloads. Quality still needs ./r9v setup ... --ced quality once for its projector.

Details: PR #42 and the changelog.

R9V v0.4.2 — opt-in CED quality mode, doctor fix after first start

R9V v0.4.2 adds an opt-in "quality" mode for the uncensored profile's fast long-prompt prefill (CED), and fixes a false doctor failure right after a first start.

What's new

• --ced quality (opt-in; in this release it fails first-start qualification on the reference host, see below; v0.4.3 fixes that) for qwen38-mtp4-uncensored. A multi-source projector that also reads the inputs of layers 3, 7, 11 and 15. In one grade of both projectors on the eager grading server, on 16 prompts that depend on long context:

  --ced	Perplexity	Long-context gain lost	Prefill speedup (median)
  on (default)	×1.049	17%	1.68×
  quality	×1.029	10%	1.55×

  It takes 1.79 GiB of VRAM per GPU (stored as int8). Run ./r9v setup ... --ced quality once: it downloads just that projector (1.8 GB), which a default setup skips. start keeps the choice.

• Doctor fix: right after a first start, ./r9v doctor --runtime no longer reports runtime-kv-pressure FAIL because of qualification's own long prompt. Any rewind after qualification still fails.

Tested on the compiled server: --ced quality does not fit here (fixed in v0.4.3)

Update: v0.4.3 makes --ced quality pass first-start qualification on this host (2.04 GiB free on GPU 0), with images still working. Use v0.4.3 for quality mode.

After release, the clean-install GPU test (fresh clone of this tag, setup --ced quality, first start on the compiled server) ran quality mode for the first time.

• It fails first-start qualification on the reference host. Every workload check ran, including the 130,941-token prompt, but quality's peak VRAM is about 0.75 GiB higher on GPU 0 and 0.35 GiB higher on GPU 1 than on. With desktop apps holding 1.24 GiB of GPU 0, GPU 0 fell to 0.90 GiB free (target 1.5 GiB; GPU 1: 1.77 GiB), so start exits with an error. on passed in the same session with 1.65 / 2.12 GiB free. Quality can only pass if other programs use less than about 0.6 GiB of GPU 0.

• Where it ran, it behaved as designed (same session, vs exact prefill):

  	quality	on
  Prefill speedup, 12,900 tokens	1.49×	1.54×
  Prefill speedup, 32,253 tokens	1.64×	1.74×
  Warm decode (ms/step, 3 runs)	38.4–39.1	39.0–41.6
  MTP tokens/step after a CED prefill vs exact	−6%	−7%

  An exact request after a CED request was bitwise identical to a fresh exact run, repeated greedy runs were identical, and nothing recompiled after the first CED request, in both modes.

• The rest of this release passed the same test: --ced on and --ced off each qualified, an unchanged restart reused its receipt, and right after on's first start ./r9v doctor --runtime reports the qualification rewinds as a NOTE instead of a FAIL (the fix above).

On v0.4.2, keep --ced on (the default); for quality mode use v0.4.3. Details: PR #41 and the changelog.

Upgrading from v0.4.1

• The runtime descriptor changed, so an existing install qualifies again once on its next start (a few minutes).
• The model package moves to revision 8112610745a8ddc3a19cc659314af245820ee728. The 24 files published before are unchanged, so nothing new downloads unless you pick --ced quality.

Testing

CPU tests, static checks and the launch parity tests (on and off launch exactly as the GPU-tested v0.4.1 container) pass. The clean-install GPU test was run after release; see "Tested on the compiled server" above.

R9V v0.4.1 — 56.3 GiB RAM floor, doctor checks, decode-speed recording

R9V v0.4.1 makes the uncensored profile need much less RAM, with the same output, and adds doctor checks and a few small fixes.

What's new

• RAM floor 71.4 → 56.3 GiB for qwen38-mtp4-uncensored. The second GPU now keeps its 400 most-used experts per layer in VRAM for good, so the host (RAM) copy of the experts shrinks from 55.4 GiB to 40.3 GiB. Output is bitwise identical to v0.4.0's runtime and decode speed is unchanged. Peak shared host memory measured at 41.7 GiB (was about 56).
• New doctor checks: free disk space for fetch and setup, VRAM held by other programs, API exposed beyond this machine, overlay file integrity, the CED projector file, and expert-limit consistency. The RAM check now names the floor it used.
• Decode-speed recording: ./r9v soak PROFILE --state-dir DIR -- --decode-speed FILE saves decode ms/step right after start and once warm.
• Longer start timeout: start now waits up to 2,400 s by default for the full expert cache.
• Image bundle download/verify: tools/image_bundle.py --verify-only downloads and checks a bundle without loading it.
• Fix: a -- after options such as --state-dir is no longer passed on to the tool (before, soak ... -- --decode-speed FILE and doctor ... -- --runtime failed with "unrecognized arguments").

Upgrading from v0.4.0

The runtime changed, so an existing install qualifies again once on its first start (a few minutes). Later restarts reuse the new receipt. No re-download of the model is needed.

Measured on the reference host (2× R9700, 128 GiB, clean install)

• Cold first start ready in 387 s; qualification passed all 7 checks, including a 130,941-token prompt.
• Peak shared host memory 41.7 GiB.
• Warm decode 36.8 ms/step on the soak prompt mix (short prose about 35.8).
• CED prefill 1.52× at 13K tokens, 1.82× at 32K. An exact request after a CED request is bitwise identical to a fresh exact run.
• --ced off start and an unchanged restart (177 s, receipt reused, doctor 0 FAIL) passed.

Known issue

Right after a first start, ./r9v doctor --runtime can report runtime-kv-pressure FAIL. It is caused by qualification's own 130,941-token prompt being preempted a few times (it still completes and passes). After a restart the doctor is clean. A fix is planned for v0.4.2.

R9V v0.4.0 — uncensored IQ4_XS, consolidated 1.3.0 runtime with CED, API on localhost

R9V v0.4.0 adds an uncensored IQ4_XS profile on the consolidated 1.3.0 runtime, with CED long-prompt prefill on by default, and publishes the API on 127.0.0.1 only in every profile.

What's new

• qwen38-mtp4-uncensored (experimental). orcarouter's abliterated Qwen3.8 Flash Next as a UD-IQ4_XS package: Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS at revision b06687cb2f83ea38039cadd249341a7bd5b76fa3, about 92.4 GiB including the CED projector. This model's refusal behavior was removed. It will comply with harmful requests. Add your own moderation before anyone else uses it.
• Consolidated 1.3.0 runtime (qwen38-flash-next-gfx1201-mtp4-v3). The existing image sha256:2dac17a2… from v0.3.0-rc1-images plus SHA-256-pinned overlays: nine Python files, the CED model file and four kernels, with their sources. ./r9v validate and the launcher refuse a modified, missing or extra overlay. No new image to download.
• CED on by default for the uncensored profile. On prompts of 8,192 tokens or more, layers 0–15 run exactly and a split-16 projector stands in for the later layers on all but the last ~2K prompt tokens. Decode stays exact. Tradeoffs, measured on the reference host:
  • about 1.5× faster prefill at ~13K tokens, rising to about 1.8× at 32K and above;
  • about ×1.051 perplexity on prompts that depend on their long context;
  • about 10% fewer MTP tokens per step on the first answer after a CED prefill;
  • 1.76 GiB VRAM per GPU, so the profile's free-VRAM target is 1.5 GiB per card instead of 3.
• --ced on|off on setup or start turns CED on or off for the server; the choice is saved and the next start qualifies again. One request stays exact with "vllm_xargs": {"r9v_ced": false}. Prompts with images and prompt-logprob requests always run exactly.
• Fixed placement for the uncensored profile: the mutable expert cache needs its pinned manifest, so --headroom, --calibration and --expert-catalog are refused. First start qualifies it once; an unchanged restart reuses the receipt.
• API bound to 127.0.0.1, all profiles. The API has no authentication. It used to be published on every interface; now it is published on 127.0.0.1 unless you set R9V_HOST_BIND when you run setup (any IPv4/IPv6 address; 0.0.0.0 for all interfaces). Bad values are refused before anything starts. If you reached R9V from another machine, set R9V_HOST_BIND and run setup again. Endpoint: http://127.0.0.1:8004/v1, health at /health.

Install

./r9v setup qwen38-mtp4-uncensored --model-dir /path/to/qwen-uncensored \
  --state-dir /path/to/r9v-state/uncensored --accept-model-license
./r9v start qwen38-mtp4-uncensored --state-dir /path/to/r9v-state/uncensored --timeout 2400

Setup downloads and hash-verifies the package, extracts the PLE table, loads the pinned image and checks the host. Installation guide · Qualification note · CED projector

Clean-install test

On the reference host (2× Radeon AI PRO R9700, 128 GiB RAM), from a fresh recursive clone with new model, data, cache and state directories, and the deployed service stopped:

Step	Result
Fetch + verify	all 20 package artifacts downloaded without a token and SHA-256 verified
Setup	passed, 133 s
First start, CED on, cold compile	ready in 337 s; qualification passed, including a 130,941-token prompt at 131,072 context; minimum free VRAM 1.771 / 2.087 GiB (target 1.5 / 1.5)
CED on the running server	projector loaded on both GPUs; 1.47× prefill at 12,960 tokens (exact tail rounds up to 3,264 there), 1.80× at 31,987; an exact request after a CED request matched a fresh exact run bit for bit
--ced off	qualified again and passed; minimum free VRAM 3.824 / 4.115 GiB
Unchanged restart	reused its receipt, ready in 167 s

The test ran at commit 34285b2; later commits change only the published address and documentation. Details: uncensored-public-userflow-20260923.json below.

Known limitations

• Experimental. Qualified on the reference host only; each new machine qualifies on its own first start.
• Decode warm-up after start. The first ~1,200 decode tokens after a fresh start run slower while caches warm up (about 45–49 ms/step with CED on, 38–41 with CED off). Warm decode with CED on measured 34.3 ms/step, matching the deployed service, where loading the projector showed no steady-state decode cost.
• CED was fitted on English-heavy code, docs and prose up to ~20K tokens; only a 2,048-token exact tail was graded; no task-level accuracy tests. int8 projector precision was never run on the GPU.
• Measured with one request at a time.
• The IQ4_XS target was quantized from orcarouter's Q8_0 with Unsloth's UD-IQ4_XS recipe; its quality was not measured against Unsloth's own UD-IQ4_XS.

Assets

The complete source archive includes the exact pinned submodule revisions. release-manifest.json and SHA256SUMS bind the release files; the runtime image is the existing v0.3.0-rc1-images bundle.

WMMA prefill IQ4 runtime bundle (image sha256:2dac17a2)

Candidate runtime image for the R9V IQ4_XS qwen38-mtp4 profile with the gfx12 int8 WMMA grouped MoE prefill kernel and 4096-token prompt chunks.

• IQ4_XS image: sha256:2dac17a215fb5b0e3461e4c3e36a2981eec8ac3d6021e73183d247e819740c03 (overlay on the public image7 sha256:46ab688af195643e61322a72b4e7b7fa0999c12299bffb2a4515f8363c59393c; the kernel is compiled in-image from source with the pinned toolchain, docker/Dockerfile.wmma-prefill).
• The Q4_K_XL profile keeps image6 from the v0.2.0-rc2-images bundle.

The image passed route capture, calibration, the planner placement and the derived full-128K workload qualification, then a fresh public setup/start with 3 GiB per-card headroom and the published prompt-processing procedure: 1,684.6 / 1,640.6 / 1,605.1 mean prompt tokens/s at 8K/32K/64K (v0.2.0: 989.3 / 982.0 / 968.2). Details and raw trials: docs/qualification/wmma-prefill-20260914.md.

The Docker save bundle (11 parts, 11,355,436,353 bytes) includes runtime dependencies and compiled kernels; release/image-bundle-wmma-prefill-20260915.json pins each part, the combined SHA-256 and the image identity. Model weights, private logs and compile caches are excluded; models are fetched separately by setup. This is an artifact prerelease, not a clean-install qualification claim.

🤖 Generated with Claude Code

R9V v0.2.0 — qualified dual-R9700 setup and restart

R9V v0.2.0 publishes the qualified IQ4_XS and Q4_K_XL setup/start/restart workflow for two Radeon AI PRO R9700 GPUs.

• IQ4 uses exact-sized cold-host allocations in image7, a measured expert catalog and its verified memory seed. Q4 retains image6 and its own placement.
• Both ordinary first starts passed all seven text, tool, vision, context and idle-resume checks, including a 130,941-token prompt at a 131,072 context limit with requested 3 GiB free VRAM per GPU.
• Unchanged restarts reused verified receipts. Both starts stopped cleanly and completed GPU reclamation and post-stop observation.
• The README includes direct model/shard and auxiliary download links. The complete source archive includes the exact public submodule revisions; SHA256SUMS and release-manifest.json bind the release files and existing runtime image assets.

Installation guide · Model downloads · Runtime images · Qualification details

Both profiles remain experimental and require local qualification on a new machine or changed placement. The verified user flows used a new anonymous source checkout and setup state with previously public-downloaded, hash-verified assets. The separate anonymous image import recovered from an interrupted load using unchanged downloaded parts. Q4 recorded one low Normal-zone sample without triggering the unchanged consecutive-sample guard; no wide host-memory margin is claimed. Original model, head and PLE bytes are unchanged.

Image7 exact-host allocator candidate and image6 runtime bundle

Candidate runtime images for the R9V IQ4_XS and Q4_K_XL release work.

• IQ4_XS candidate image7: sha256:46ab688af195643e61322a72b4e7b7fa0999c12299bffb2a4515f8363c59393c
• Existing Q4_K_XL image6: sha256:2e50016cfcc9cd22f15d3f69ccf001e4877236e12ebb4ab458cc9c16caaef9e3

Image7 avoids power-of-two rounding for streaming cold expert host allocations. The exact-size path preserves the existing HIP default flags and tensor ownership. Implementation and overlay Dockerfile: 123f1ad.

Image7 passed real packed-storage parity, a fixed-placement full128K workload with 3GiB requested headroom per GPU, and a second startup using the same compiled cache. Public setup/start/restart qualification, refreshed placement maps, and BetterBench results are still pending. This artifact prerelease is not the final source release or a clean-install qualification claim.

The complete Docker save bundle includes runtime dependencies and compiled kernels. The manifest pins each part, combined SHA-256 and both image identities. Model weights, private logs and user compile caches are excluded. Models are fetched separately. The existing default setup remains unchanged until the candidate profile is qualified and published.

Docker 29 with the containerd image store preserves these OCI index identities. Retain and verify every ordered part against public-manifest.json before loading. The public content review records the exact base and added source files.

R9V MTP4 runtime images — IQ4_XS and Q4_K_XL

Exact R9V MTP4 runtime images for the dual-R9700 IQ4_XS and Q4_K_XL profiles.

The gzip-compressed Docker save bundle contains both images and their shared layers. It is split into files below GitHub's per-asset size limit. The manifest pins every part's length and SHA-256, the ordered combined archive hash, and both image identities.

Image identities:

• IQ4_XS: sha256:987468f3f9991dfad8b07f51a18bbd5e5c01dc164d9f82c4143e77bfd14ca80d
• Q4_K_XL: sha256:2e50016cfcc9cd22f15d3f69ccf001e4877236e12ebb4ab458cc9c16caaef9e3

These are the tested images, including the runtime dependencies and compiled kernels. Model weights, user state, private session logs and compilation caches are not included. Model packages are fetched separately at their pinned revisions.

Use the current R9V setup command, which verifies all parts before loading the selected image. Docker 29 with the containerd image store preserves these OCI image-index identities. This image artifact release is separate from the final source release and clean-install/BetterBench qualification report.

R9V Qwen3.8 Flash Next v0.1.0

First public R9V release for Qwen3.8 Flash Next on dual gfx1201/RDNA4.

Highlights:

• OpenAI-compatible text, tools, and vision serving
• TP2 with IQ4_XS target, Q8 vision projector, and block-FP8 MTP2
• 128K context profile with SSD-backed 51B PLE table
• Shape-specialized dense, tiered MoE, and speculative GDN HIP kernels
• Measured sustained TG: 77.82-79.27 tok/s from 8K through 120K context on the reference dual-R9700 system
• Reproducible pinned vLLM/plugin/kernel sources and container build

Model weights remain under the Qwen Community License and will be published separately on Hugging Face.