Penny Royal 3.1.2

Penny Royal 3.1.2

This patch release gives the maintained native-MTP profiles one public
width-policy setting, fixes DFlash2 decode-checkpoint tracking, stops the
scheduler's idle sleep from delaying encoder-embedding handoffs, and labels
both DFlash2 profiles experimental / alpha. Model, profile and performance
defaults are unchanged from 3.1.1: the tiered width routing still launches
exactly what it launched before.

Published image

Image: ghcr.io/jpezzulli/sglang-rtxpro6000:v3.1.2

Digest: sha256:013a685d042b178842cbf016e7deb8d23113b87044355ef785dbee6d56e57db6

The image was built from fe35e1d817 and promoted after checking, without rebuilding. The release tag also includes final publication documentation.

Fixes and changes

• One width-policy setting. The native-MTP launch recipes hard-coded one
  policy file and a checksum that refused anything else, so returning to the
  3.0.2 acceptance-adaptive widths meant editing launch lines. PENNY_MTP_POLICY
  is now the single setting that selects both the bundled policy file and the
  launch width that can carry it: tiered (the unchanged default — block 8 at
  one running request, block 4 at two or more), adaptive (the v3.0.2 policy
  restored byte for byte), or fixed-w4 (block 4 at every concurrency).
  Unset keeps the default, so an existing launch is untouched. The checksum
  gate is gone: the selected policy file's hash is now NIXL cache identity,
  not a boot refusal. The setting does not apply to the DFlash2 and 27B
  profiles, which route their own widths
  (#118).
• DFlash2 decode checkpoints. The checkpoint-crossing check compared
  pre-verify lengths against themselves, so the mask was always false and the
  backend received "no tracking" even when an accepted draft block clearly
  crossed the grid — while the live state still committed, letting result
  processing register or promote a checkpoint that was never written. The
  post-verify lengths are now derived from the commit lengths. Both DFlash
  profiles were affected
  (#115,
  #119).
• EPD idle sleep. With --sleep-on-idle and language-only EPD separation,
  a request waiting for its encoder embeddings left the scheduler "idle", and
  the sleeper's poll did not watch the per-request embedding socket, so the
  request waited for the 1-second poll timeout. The scheduler now skips the
  sleep while embeddings are waiting
  (#117).

DFlash2 is experimental / alpha

Penny's DFlash2 integration — both the Flash-Next next-dflash2 profile and
the 27b profile — is now labelled experimental / alpha at every
user-facing surface. This is a maturity label, not a toggle and not an
automatic fallback to native MTP; the profiles stay fully available to opt
into, and the maintained default stays native NEXTN MTP.

What that label means today, in two separate parts:

• Known and documented, not fixed here: the DFlash2 draft keeps its own
  host cache in addition to the target's PENNY_HICACHE_SIZE_GB budget.
  This extra allocation is observed, not theoretical: locally, a 32 GB target
  HiCache pool plus about 22 GB of draft cache is roughly 54 GB of caches
  alone before the PLE table and other process memory, and one contributor
  reported a 48 GB target HiCache budget plus a further 32.9 GB of draft
  cache with the host swapping. The actual amount scales with the target's
  logical slots and the draft's geometry, so size host RAM from what the boot
  reports rather than from the HiCache setting alone. 3.1.2 documents this
  bill; it does not fix or reduce the allocation. If DFlash2 demonstrates a
  speed advantage over native MTP, we’ll work on reducing this extra memory
  use.
• Not yet validated: the #115 fix is covered by a CPU regression over
  both profiles' checkpoint grids, but the on-GPU state scatter and an actual
  cache-resume were not exercised by it, on either profile.

Update

Native users can follow BUILD.md. Container users should pull
ghcr.io/jpezzulli/sglang-rtxpro6000:v3.1.2 and restart with their existing
external configuration. New host startup scripts and image v3.1.2 are a pair:
the 3.1.2 startup scripts require the mtp-policy.sh helper that only ships
in the v3.1.2 image, so replace your downloaded launch files and move to the
new image together (container guide).
Fresh launch files and the beta configurator select the new image, but an
explicit older image pin in saved settings survives regeneration on purpose:
changing the default does not move your pin, so update it to v3.1.2
yourself when you upgrade. The public default launch is unchanged from
3.1.1; adaptive and fixed-w4 are operator choices, not new defaults. The
3.1 release notes contain the feature list and
performance results.

Thanks

• Jim Smith (jhsmith409) — the
  encoder-embedding idle-sleep fix (PR #117).
• Lesilva — the #115 report, proposed
  arithmetic and runtime evidence.
• Penny Royal — width-policy setting,
  checkpoint fix, alpha labeling.

Validation

Run on this candidate source (2026-10-23), CPU only:

• 20 focused config/setup/recipe/startup-summary tests with 33 subtests
  passed (policy selection, image defaults, release asset, generated and
  shipped launch lines; 170 unrelated cases deselected).
• 65 registered tests passed across the prebuilt-container launch checks
  (shell-to-recorded-server policy selection, parity, saved settings,
  namespace identity, invalid selections, DFlash2 isolation), container
  release routing, the startup version banner and the DFlash2
  checkpoint-tracking regression.
• The checkpoint regression is red on the release base and green with the
  fix. Pinned CI formatting (Black 26.1.0) passes on all changed files.

The published image passed its CPU package check and checks of its installed
runtime fixes, policy assets and NVMe reader package. Default, tiered, adaptive
and fixed-W4 selections were exercised inside the image with GPUs hidden.
Native Flash-Next NEXTN MTP was restarted with adaptive selected and passed
real chat and structured-tool smoke checks.

No DFlash2 GPU checkpoint/cache-resume validation, GPU container-serving test,
NVMe serving test or benchmark was run for this patch. The reusable runtime
qualification is the dated 3.1 evidence in release-3.1.json and RESULTS.md;
3.1.1 supplied CPU/static and image-package verification. Public profile and
performance defaults are unchanged, and measured results remain dated.
See release-3.1.2.json for the exact image identity,
build, promotion and verification record.

Penny Royal 3.1.1 — NVMe PLE startup fix

Penny Royal 3.1.1

This patch release fixes one startup problem reported after 3.1: explicit
NVMe PLE selection was rejected at launch.

Fix

• NVMe PLE startup. The bundled NVMe reader checked whole-module hashes
  of the runtime source it registers against, and those hashes were still
  the ones taken on the v3.0.1 tree. Any later source edit — including the
  unrelated changes that shipped in 3.1.0 — made the comparison fail, so the
  reader refused to register and NVMe PLE startup was rejected before the
  model could load. The whole-module gate and its source manifest are removed
  (#111).

Everything else about the reader is unchanged: table and artifact validation,
cache identity, the requirement to select NVMe explicitly, and the failure
handling for missing required hooks all remain enforced. Model, profile and
performance defaults are unchanged from 3.1.

Update

Native users can follow BUILD.md. Container users should pull
ghcr.io/jpezzulli/sglang-rtxpro6000:v3.1.1 and restart with their existing
external configuration. Fresh launch files and the beta configurator select
the new image; an explicit older image pin in saved settings survives until
you change it. The 3.1 release notes contain the
feature list and performance results.

Thanks

• @desva — NVMe PLE startup failure report.
• Penny Royal — fix implementation.

Validation

75 CPU tests and 9 subtests passed. The release image passed its package
checks and contains the corrected reader with the obsolete manifest removed.
Source, image digest and build details are in
release-3.1.1.json.

Penny Royal 3.1 — native MTP and optional DFlash2

Penny Royal 3.1

Penny Royal 3.1 keeps native NEXTN MTP with FR-Spec as the maintained default
for Qwen3.8 Flash-Next, with its verify width fixed by active concurrency
(block 8 for one running request, block 4 for two or more). Flash-Next DFlash2
with FR-Spec is fully available as an optional profile — same weights, same
cache stack, its own draft model — now verifying draft tokens with
FlashInfer's SM120 kernel. The release also replaces the preset KV-token cap
with automatic pool sizing. Qwen3.8-27B FP8 with DFlash2 remains the second
supported profile, unchanged. The target is RTX PRO 6000 Blackwell with one
card (TP1) or two (TP2).

Flash-Next keeps native MTP as the maintained default

The next profile (FR-Spec) and next-plain (without FR-Spec) keep their
existing recipes — .97 and .981 static memory fraction, 4 requests /
24 Mamba slots unless you override them — and keep receiving updates and
testing. New setups start on next, both native profiles remain maintained,
and a saved explicit choice is always preserved.

The native verify width is fixed by active concurrency: block 8 for a
single running request, block 4 for every batch of two or more, read from the
bundled policy configs/pennyroyal/adaptive-next.json. next-plain uses the
same widths without the FR-Spec map. A single request stays at width 8
instead of switching widths with acceptance feedback.

Flash-Next gets its own DFlash2 draft

The optional profile (PENNYROYAL_PROFILE=next-dflash2) runs the accepted
OrcaRouter ModelOpt NVFP4 Flash-Next target — same weights, same precision,
same cache stack — against the separate
incoai/Qwen3.8-Flash-Next-DFlash2
draft at full 524,288-token context. That draft is the one new download this
release, needed only for the optional profile. The profile ships with the 65,536-ID FR-Spec map, the resident
rowwise-FP8 output head shared with the target (PR90), candidate projection
through the FR-Spec map (PR93), a .95 static memory fraction for
image-workload headroom, 6 requests / 36 Mamba slots, a 2,048-token draft
window, and draft GPU KV bounded to the sliding window without taking host
HiCache capacity (PR91, PR92).

The profile now uses FlashInfer for target linear-attention verification
(--linear-attn-verify-backend flashinfer, PR104). Both the native launch
files and the container startup files select it automatically. On the final
candidate, matched screens show single-request decode at block 6 rising from
227.37 to 241.49 tok/s mean (sampled peak 339.53) and the full six-request
batch at block 4 from 503.41 to 504.80 tok/s aggregate (peak 568.90); the
completed ten-slide presentation reached 217.19 tok/s mean (peak 448.70).

Verification widths are fixed by active concurrency: block 6 for a single
running request, block 4 for every batch of two or more, read from the bundled
policy configs/pennyroyal/adaptive-dflash2.json.

DFlash2 offers more token-pool capacity; we are still evaluating performance
across concurrency levels. TP2 is expected to work for this profile; it has
not been tested on two-GPU hardware.

KV pool sizing is automatic

The launch recipes no longer request a preset KV-token cap (PR86). SGLang
sizes the shared pool from the GPU memory it profiles at boot, inside the
recipe's static memory fraction, and reports the resolved number in the startup
output; this system resolved a 1,210,944-token target KV pool for the optional
DFlash2 profile and a 1,081,792-token pool for the final native-MTP
measurements. The maintained native-MTP profiles
size their pool the same way and report their own resolved number at startup.
An explicit MAX_TOTAL_TOKENS remains available as a deliberate,
page-aligned positive cap on every profile, and startup summaries say which of
the two paths produced the number you are reading.

Fixes

• Finished requests are dropped from the running batch before every admission,
  and a request's held req_to_token row is counted once (PR87, PR88), so
  admission decisions match the batch that is actually running.
• Stop-regex matches inside accepted speculative blocks are honoured on both
  DFlash2 block sizes (PR96).
• Mixing token-probability requests with ordinary generation no longer
  triggers the identified scheduler crash (PR97).
• The beta configurator learned the next-dflash2 profile on both the native
  and container paths (PR94): it prompts for the draft checkpoint and carries
  the fixed-width policy and FlashInfer verification from the samples. The
  external container configuration files and the manual native recipes remain
  the primary setup paths; the configurator stays an optional beta.
• The agreed upstream-fix subset (PR79) lands QSA cache-restore ordering and
  packed-scratch initialization, the shared Chat/Responses chat_template
  trust gate, absent tool_calls fields, stable stream timestamps, 400
  responses for truncated or corrupt images, collection before KV sizing,
  ignore_eos token reservation, and freed-response-state handling.
• The optional Swift FP8 PLE transplant/verification recipe (PR76) validates
  the whole safetensors data section, table widths, scales and payload layout
  before staging anything; its serving qualification remains contributor-owned.

Current results

SGLang generation rates on one RTX PRO 6000: mean and sampled peak, including
reasoning, with normal non-greedy sampling. The final native MTP runs used
block 8 at C1 and block 4 at C4/C6. Full settings and source records are in
RESULTS.md.

Profile / workload	Mean tok/s	Sampled peak tok/s
Flash-Next native MTP — single request, C1 W8, fe5f02c (maintained default)	197.87	345.30
Flash-Next native MTP — full four-request batch, aggregate, W4, fe5f02c	555.11	731.06
Flash-Next native MTP — full six-request batch, aggregate, W4, fe5f02c	472.79	528.49
Flash-Next native MTP — coding workflow (agentic), C1 W8, medium, fe5f02c	252.83	501.63
Flash-Next native MTP — presentation workflow (agentic), C1 W8, medium, fe5f02c	216.23	525.80
Flash-Next DFlash2 + FlashInfer — single request, C1 W6, a60f442 (optional profile)	241.49	339.53
Flash-Next DFlash2 + FlashInfer — full six-request batch, aggregate, W4, a60f442	504.80	568.90
Flash-Next DFlash2 + FlashInfer — presentation workflow (agentic), C1 W6, explicit medium, a60f442	217.19	448.70
27B DFlash2 — single request, fixed W8, 27f9eaa	133.06	232.07
27B DFlash2 — full four-request batch, aggregate, 27f9eaa	459.74	570.73

Cold prompt	Flash-Next DFlash2 prefill tok/s	27B prefill tok/s
63,906 tokens	15,365.38	6,421.59
489,921 tokens	8,715.05	1,677.10

The native coding workflow completed naturally at 27 model turns and the
agentic ten-slide presentation at 80 model turns, both at medium reasoning on
block 8. The DFlash2 reasoning qualification scored 93.46/100 for Flash-Next
and 96.94/100 for 27B, recorded on the DFlash2 profile.
RESULTS.md records the qualification and all
measurements.

Updating

Native: follow BUILD.md with
RELEASE_REF='pennyroyal-v3.1.0'. Container:
download the release's launch-file set and run
ghcr.io/jpezzulli/sglang-rtxpro6000:v3.1.0 with the matching startup files,
per the container guide. Update any explicit
image pin — PENNYROYAL_IMAGE= in saved setup settings or the Compose .env,
or the IMAGE= value in an edited run.sh — to
ghcr.io/jpezzulli/sglang-rtxpro6000:v3.1.0, or pass that image name with --image. Existing target weights
are unchanged; the Flash-Next DFlash2 profile needs the new
incoai/Qwen3.8-Flash-Next-DFlash2 draft download. Your saved settings, model
folders and persistent caches stay as they are, and no blanket cache deletion
is required. Regenerate setup files with the 3.1 configurator or edit the 3.1
samples; every save writes new timestamped files and preserves existing ones.
The complete settings list, defaults, interactions and restart/cache effects
are in CONFIGURATION.md.

Thanks

• @jhsmith409 and
  @LocalhostRonin — direct Penny Royal
  contributors to this release.
• Penny Royal did almost all of the
  coding, including the ports and integration work.
• John, Penny and Codex worked through the
  architecture, tuning and debugging together — including the width-routing
  decision, the memory-fraction measurements and the verification-backend
  choice.

Source: pennyroyal-v3.1.0.
Container: ghcr.io/jpezzulli/sglang-rtxpro6000:v3.1.0.
The exact image digest and build are recorded in
release-3.1.json.

Penny Royal 3.0.2 — lighter caching and targeted fixes

Container: The 3.0.2 image is building. This page will be updated when the download is ready.

Penny Royal 3.0.2

This maintenance release reduces host-memory overhead in persistent caching and brings focused improvements to speculative decoding and client compatibility for both supported model profiles.

Released October 17, 2026. Container build status is recorded on the GitHub release page.

What changes

• Less host RAM used by NIXL caching. Mamba host-state slots are now page-aligned, allowing direct storage I/O to use the existing host pool without two redundant bounce buffers. In the measured Flash-Next configuration, this recovered about 14.1 GiB of host RAM. Thanks to Kaspar Holmgren (@hiyazakite) for PR73.
• Less rapid MTP width switching on Flash-Next. The existing acceptance-based selector continues learning while a hold period limits repeated switches between four- and eight-token speculation for a single request. The hold starts at 40 completed feedback batches and backs off to 80 after repeated reversals. Two or more concurrent requests still select four-token speculation immediately.
• Complete compact draft-attention windows on 27B. DFlash2 now includes the verification block when calculating the host-side attention length after applying the window limit. This keeps the draft's intended attention window complete.
• Correct grammar-mask handling on 27B. DFlash2 clears stale request-level grammar masks before speculative verification. The verifier continues to apply the grammar constraints for each token position.
• Better assistant-history compatibility. Both profiles accept the reasoning field when reasoning_content is absent or null. An explicitly supplied reasoning_content string, including an empty string, retains precedence.

Validation and model settings

The profiles remain Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with NEXTN MTP, both using HiCache/NIXL. Model weights, quantization, default precision and configured capacities are unchanged.

Local single-GPU qualification covered both profiles: reasoning, 30/30 streaming tool checks each, images, 489,921-token cold prefill and recall, and disk-restart restoration of KV plus Mamba state.

Updating

Follow the native build instructions or container guide. Use ghcr.io/jpezzulli/sglang-rtxpro6000:v3.0.2 with the matching 3.0.2 startup recipes or configurator output, preserving your paths and settings. Custom startup scripts that pin the adaptive-policy hash need the matching updated recipe. This release does not require downloading model weights again or deleting persistent caches.

Thanks to Kaspar Holmgren for the original NIXL contribution and Penny (Pennyroyal@agentmail.to) for the accompanying fixes.

Penny Royal 3.0.1

Penny Royal 3.0.1

This maintenance release fixes three startup problems reported after 3.0 and
adds the Penny Royal version to startup logs.

Fixes

• RAM PLE on WSL2. Both PLE gather paths now use the device address returned
  by cudaHostGetDevicePointer. WSL2 can map registered host memory at a
  different address; passing the host address caused an illegal memory access
  on the first request. The address is resolved once per allocation, with
  cleanup if resolution fails. (#64)
• Persistent TileLang cache. Compiled kernels now go under the configured
  cache directory, normally /cache/tilelang in the container. This fixes
  startup with custom user IDs and keeps the cache across container recreation.
  An explicit TILELANG_CACHE_DIR is preserved.
  (#65)
• NVMe PLE startup. The reader's compatibility hashes now match the shipped
  source. The stale 2.5.3 hashes rejected the 3.0 attention backend before the
  model could load. Required hooks and mismatch checks remain enforced.
  (#66)

Startup version

The launchers now print Penny Royal 3.0.1 and the source commit. Locally
modified source is marked in the same line, making bug reports easier to trace.

Update

Native users can follow BUILD.md. Container users should pull
ghcr.io/jpezzulli/sglang-rtxpro6000:v3.0.1 and restart with their existing
external configuration. Model and cache mounts stay in place. Fresh launch
files and the beta configurator select the new image.

The 3.0 release notes contain the feature list and
performance results. These fixes keep the same model and performance defaults.

Thanks

• alphastorm — WSL2 reproduction, device-pointer
  fix and testing.
• Matt Sapp — custom-UID failure and persistent
  TileLang cache fix.
• desva — NVMe PLE compatibility report.

Penny Royal implemented the fixes; Codex reviewed and checked the integration.

Penny Royal 3.0 — Faster agents, adaptive MTP, simpler setup

Penny Royal 3.0

Released October 10, 2026.

Penny Royal 3.0 brings adaptive speculative decoding, faster small-batch
kernels, and more room for conversation state to Qwen3.8 Flash-Next. The target
is RTX PRO 6000 Blackwell with one card (TP1) or two (TP2). Qwen3.8-27B FP8
with DFlash2 remains the second supported profile.

Faster Flash-Next generation

• Adaptive MTP is enabled by default, and adapts only at C=1. One active
  request switches between four- and eight-token windows according to draft
  acceptance. At C=2 and above, draft width is fixed at four; speculative
  decoding continues. A server configured for C6 can still adapt while only
  one request is active. C counts requests, independently of TP1/TP2 GPU count.
  Per-width graph state and target tuning preserve state when the width changes.
  FR-Spec stays enabled at every concurrency: it narrows the draft vocabulary,
  while adaptive MTP selects draft length. The target keeps its full vocabulary.
• FP8 and small-batch kernels select themselves. Eligible SM120 launches
  use rowwise-FP8 dense weights and specialized projection/output-head kernels
  that read those weights directly. No copied benchmark flags are needed.
• Less work in each draft step. Dedicated HyperConnection mixing, BF16 MTP
  entry projection, NVFP4 draft-MoE, and packed-key routing kernels reduce work
  repeated for every generated token. The router preserves expert selection.
• GDN improvements. Fused normalization/output projection removes an
  intermediate memory round trip. Extend/prefill computes gating beta in FP32
  to avoid the previous BF16 rounding.

More room for context

Attention setup no longer constructs a backend that QSA immediately replaces,
and compatible QSA decode paths share scratch space. These savings recovered
adaptive MTP's extra memory cost. On our one-card C6 setup, the GPU token pool
is 1,067,072 tokens, with 4.194 GiB free at startup.

The pool is shared across requests and cached prefixes. The per-request
context limit remains 524,288 tokens.

Fixes for agents and long sessions

• Reserve speculative Mamba buffers from their actual allocation requirements,
  leave context room for speculative lookahead, and retain state at the
  committed-token boundary.
• Keep useful host Mamba checkpoints, release abandoned session branches, and
  track shared-backup ownership. Session retention uses the client's session
  IDs and close calls.
• Preserve literal thinking tags in answer text after reasoning ends.
• Emit Responses reasoning-part events and report capped or failed turns with
  the correct terminal status.
• Count tool-parser outcomes in Prometheus without logging prompts or arguments.
• Reject an image request whose shared-memory segment has disappeared, release
  its feature buffers, and keep the scheduler serving other requests.
• Separate persistent cache namespaces for the effective FP8 representation
  and GDN mode, and correct WSL2 host-memory aliases and hybrid cache keys.

Simpler installation and a slimmer release

• Configuration lives outside the container. Your host run.sh chooses the
  image, GPUs, ports and mounts. Your host startup file holds model and SGLang
  settings. Docker mounts that configuration directory read-only at /config,
  and the container runs the mounted file. Models and caches have separate
  mounts. Edit and restart to change settings; keep the same files when updating
  the image. No image rebuild or Compose setup is required.
• Optional NIXL disk caching. It stays on by default. Turning it off keeps
  GPU and RAM caching; PLE placement is independent.
• Beta configurator. Generates ordinary timestamped launch files, leaves
  previous files untouched, and prints the command for the files just saved.
• FlashInfer 0.7.0 post1. Matching CUDA 13 providers and the accepted SM120
  MoE/GDN sources are included. Interrupted native-extension builds recover on
  the next run, and build failures retain compiler diagnostics.
• The release went on a diet. We kept RTX Blackwell for RTX PRO 6000
  inference and Ampere/Ada Lovelace for image-processing sidecars, and
  trimmed the other GPU build targets. A slimmer release for easier downloads.
  CPU preprocessing remains available.

Performance on one RTX PRO 6000

OrcaRouter Qwen3.8 Flash-Next ModelOpt NVFP4, FR-Spec plus adaptive MTP, 524K context limit,
normal non-greedy sampling. Rates come from SGLang's native generation logs.

Workload	Mean tok/s	Sampled peak tok/s
Real coding workflow	262.32	518.18
Presentation workflow	239.53	543.72
Single request	218.63	326.56
Four concurrent requests, aggregate	489.35	636.71
Six concurrent requests, aggregate	433.54	606.48

Reasoning is included. Prefill, tools, and idle time are measured separately.
The peaks come from the same measured windows as the means.

Cold prompt	Mean input tok/s	Time to first token
64K	17,235.90	4.71 s
490K	8,839.75	60.37 s

Recorded improvement over 2.5.3

The saved native regular C1 mean rose from 186.79 to 218.63 tok/s (+17.0%),
and the C6 GPU token pool grew 2.7%, from 1,039,040 to 1,067,072 tokens.
The context limit remains 524,288. These are the recorded release configurations:
medium reasoning in the October 6 baseline, xhigh in the October 10 run.
Results retains the request settings.

Public code-edit score

The accepted adaptive screen reached 491.88 tok/s mean / 574.16 sampled peak
on the public code-edit prompt with normal non-greedy sampling. Our saved 2.5.3
code-edit set averaged 336.57 tok/s using greedy sampling. The 3.0 screen
used the accepted code with a 1,048,576-token pool before the final pool increase.

The prompt is a mechanical rewrite of supplied code, which gives speculative
decoding highly predictable text. It is reported separately from the real
coding workflow's 262.32 mean / 518.18 peak, which includes model turns
that use tools and write and test code. Code-edit settings
are recorded alongside the measurements.

The reasoning suite scored 95.69/100, with all nine scenarios completing.
All 30 tool responses were parseable; 28 passed automatically and two hit a
confirmed fixture-classifier error. Image input and RAM cache restoration
also passed. Detailed results and the
machine-readable release notes retain the exact conditions.

Install or update

Use the native installation guide or the
prebuilt container guide. Source tag:
pennyroyal-v3.0.0; container image:
ghcr.io/jpezzulli/sglang-rtxpro6000:v3.0.0. The prebuilt image is available now
(about 7.65 GB to download).

Thanks

• aiueo52 — Flash-Next
  optimization work and the discussions that helped us bring it into Penny.
• Tyler Landis / LandOfLemons — session
  and host-cache work, WSL2 fixes, and integration feedback.
• Martin Kazimir Kaplan / cube4elements
  — the image shared-memory crash report and fix.
• untcoder2 and
  StockSpecialist1707 —
  TP2 field testing and practical configuration feedback.
• palves — container configuration feedback
  and shared launch customizations.

Penny Royal did almost all the coding for this release,
including the ports and integration work. The engineering was a collaboration
between John, Penny and Codex. We worked through
the architecture, tuning and debugging together, right down to fused kernels,
memory layouts and milliseconds per decode cycle. Lots of back and forth
figuring out why something that should be faster wasn't, and deciding what
was actually worth keeping.

Pennyroyal v2.5.3 — Checkpoint, tool streaming and cache fixes

Pennyroyal v2.5.3

This release fixes Flash-Next’s native MTP checkpoint selection and incorporates the BF16 NVMe PLE hotfix, alongside improvements to tool streaming, prompt processing and cache persistence.

Main fixes

• Flash-Next checkpoint selection — #17: Corrects which recurrent state is saved at the accepted-token boundary during native MTP speculative decoding. The wrong boundary could leave cached state inconsistent with the accepted tokens, causing the reported phantom-token and file-writing problems. Thanks to barnesea for the analysis and hey-heiko for reproducing and checking the fix.
• BF16 NVMe PLE staging: Allocates staging memory using the PLE table’s actual element size. This fixes Swift checkpoints failing with PLE lookup needs ... staging rows when the table uses BF16. The optional v2.5.2 hotfix is now included.

Tool calling and client compatibility

• Incremental string arguments: Streams string tool arguments as they are generated instead of buffering until the closing parameter tag. Long file-writing calls can show progress throughout generation. Adapted from SGLang #41313.
• Quoted tool examples in reasoning: Fixes handling of quoted tool markup split across streaming chunks. Unknown tool calls continue to reach the client for normal error handling and recovery.
• Thinking settings: Accepts either thinking or enable_thinking for a boolean chat-template setting.

Prompt processing and caching

• Parallel prompt encoding: Tokenizes eligible long prompts in parallel while preserving the exact token IDs. Other inputs use the existing serial path. Adapted from SGLang #41259.
• Cache persistence: Ensures fresh prefill chunks are backed up and keeps their host buffers alive until pending storage writes finish, including when a cached prefix is split.
• Cache disk warnings: Warns at startup when the cache filesystem is under pressure. Old cache directories remain under your control.
• 27B/DFlash2: Uses pinned memory for small prefill metadata transfers to avoid blocking copies. Adapted from SGLang #40091.
• WSL2 HiCache fixes: Adds an optional pinned-memory allocator to address illegal memory accesses caused by WSL2’s handling of CUDA host-memory pointers. Enable it with SGLANG_HICACHE_TORCH_PINNED_ALLOC=1 before starting the server. Also corrects CUDA device pointers for Mamba transfers and NIXL hybrid storage-component counts. Adapted from LandOfLemons’ PR #24.

Testing and updating

Both supported profiles completed full reasoning and tool runs, plus 64K disk-cache restoration and replay checks. The uncapped 27B tool rerun passed all 30 cases; Flash-Next also passed all 30 tool checks.

The dependency stack and model weights are unchanged. Update the runtime or container; v2.5.3 already includes the separate BF16 PLE patch.

WSL2 setup update: The configurator now offers the host-memory workaround under Advanced, and Compose forwards it to the container. Get the updated setup files. The runtime and v2.5.3 image are unchanged.

Community thanks

Special thanks to u/StockSpecialist1707 for helping work through TP2 support, testing larger token pools, and sharing the real-world feedback that helps make Pennyroyal better.

Setup and update instructions · Full changelog and credits

Pennyroyal v2.5.2 — Memory, cache and easier setup

Pennyroyal v2.5.2 brings lower memory overhead, cache fixes, and easier setup.

• Less temporary GPU memory during Flash-Next loading and long prefills, plus exact-sized pinned RAM for the PLE table.
• HiCache reclaims only the space it needs. Choose your HiCache RAM size and set a NIXL disk-cache budget.
• Optional beta configurator with numbered choices, explanations, and saved settings for native and container installs.
• TP2 preparation, including separate NIXL namespaces and FR-Spec weight handling. TP2 verification is pending from u/StockSpecialist1707.

Both Next and 27B completed regression testing. Existing performance tables and model defaults remain.

Setup · Full changes and upstream credits

The container image ghcr.io/jpezzulli/sglang-rtxpro6000:v2.5.2 builds automatically after publication. Follow the container build.

Pennyroyal v2.5.1 — cache and parser maintenance

Pennyroyal v2.5.1 — cache and parser maintenance

This release improves long agent sessions and prefix reuse while keeping the
v2.5.0 model profiles, precision options and defaults.

• Both models: fix quoted tool markers cutting reasoning short, and keep
  fenced tool examples as text while preserving genuine calls.
• Both models: keep useful host-cache checkpoints under pressure and skip
  optional checkpoint donations when the recurrent-state pool is busy.
• Flash-Next: fix chunked-prefill handling for unaligned QSA prefixes.
• Optional C6: six concurrent Flash-Next requests with a larger shared KV
  pool, still 524,288 tokens per request. Native and Compose instructions included.
• Containers: GitHub now builds the matching image when a release is
  published. Native installation remains available.

The reported host-only HiCache case improved from 0/9 to 9/9 substantial
restores on the same workload, with median revisits dropping from 9.04 s to
1.05 s. Regression covered both models' prefill, decode, concurrency, tools,
vision and NIXL restart paths. Dependencies and existing speed tables are
unchanged.

Thanks to the reporters of #3
and #6, and the upstream
work in SGLang #37619 and
#39575.

Setup
· Full changes
· Container build status

Pennyroyal v2.5.0 — Qwen3.8 SGLang on RTX PRO 6000

Qwen3.8 SGLang on RTX PRO 6000 — Pennyroyal v2.5.0

Pennyroyal v2.5.0 is the SGLang-derived runtime for Qwen3.8-27B FP8 with DFlash2 and Qwen3.8 Flash-Next NVFP4 with FR-Spec on one NVIDIA RTX PRO 6000 Blackwell 96 GB GPU (SM120). Both profiles retain 524K context and HiCache/NIXL. This release adds two new, optional Flash-Next features:

• Online FP8 acceleration: convert selected projections, HyperConnection
  mixing weights and the output head during loading. On one RTX PRO 6000,
  measured C1 decode improved 28.3% at short context, 26.5% at 128K, and
  15.8% at 490K, retaining 524K context, 824,384 KV tokens, FR-Spec and
  HiCache/NIXL. BF16 recurrent state and NVFP4 experts remain unchanged.
• NVMe-backed PLE: stream the approximately 47.68 GiB PLE table from
  a prepared local SSD snapshot instead of pinning it in system RAM. RAM
  remains the default. This lowers fixed RAM requirements, with a documented
  performance tradeoff rather than a promise of free speed.

C1 decode workload	Option off	Online FP8
Short context, 1,024 output tokens	161.47 tok/s	207.12 tok/s
128K context, 1,024 output tokens	154.70 tok/s	195.63 tok/s
490K context, 1,024 output tokens	149.14 tok/s	172.64 tok/s

These are post-first-token rates. Short-context values are three-run medians;
long-context values are single observations. Cold TTFT did not improve.
Post-graph free VRAM was 7.52 GiB versus 3.66 GiB in the earlier matching boot:
about 3.86 GiB additional headroom, not seven GiB newly recovered.

The launcher guides explain CPU, main-GPU and secondary-GPU image
preprocessing, plus explicit KV-pool sizing. Extra media workspace and extra
KV compete for the same VRAM; do not assume every option can be combined.
This release also adds structured-output startup warmup and releases a
temporary PLE loader reference promptly. The 27B/DFlash2 profile remains
supported on the same runtime.

Direct credit to
mratsim/sglang-qwen38fn-sm120-turbo
for the online-FP8 approach and supporting startup improvements, and
Garner McCloud's SSD Stream
for the NVMe PLE reader. Upstream licenses and detailed attribution are retained.

Start with the README,
FP8 guide and limitations,
NVMe PLE setup,
and build/update instructions.
Install the release source to use these features; an older wheel alone does
not acquire the new implementation.