v0.16.0 - Continuing-turn latency and independent target replay

v0.16.0 — Continuing turns benefit from retained state and stock offload

A resident conversation changes the practical result. On the pinned RTX 5080
Laptop system, the stock threshold 8 / K16 configuration takes 34.67 seconds
for five continuing turns after the initial 16K request, versus 79.98 seconds
for retained threshold 32 / K4. Mean client time to first token is 0.766 seconds.
The configuration uses Qwen2.5-32B-Instruct Q4_K_M with a 0.5B Q8_0 draft in stock
llama.cpp b10919; no runtime patch or new controller is involved.

Configuration	State	Mean five-turn time	Mean client TTFT	Emitted IDs / whole-request second
Default threshold 32, K4	Retained	79.984 s	0.802 s	3.513
Default threshold 32, K4	Reset	246.449 s	34.708 s	1.104
Threshold 8, K16	Retained	34.669 s	0.766 s	8.221
Threshold 8, K16	Reset	199.099 s	34.087 s	1.366

These aggregates exclude the initial cold request and pool two fresh-process
repeats. Reset controls receive each retained run's exact recorded prompts.
Both configurations reuse the long prefix; operation offload remains useful
after that reuse, with approximately 2.34x continuing emitted/request throughput.
This metric includes incremental prefill and is distinct from v0.15's native
decode rate on a different sustained-generation fixture.

Inputs match on all twelve retained cross-configuration requests; outputs match
on ten, with the final topic-change answer differing in both repeats. All 24
within-configuration repeat comparisons reproduce IDs. Retained/reset outputs
match on 18/24 full requests, including 14/20 continuing turns. The table therefore
reports measured same-input latency and actual emitted counts, not universal
identical-output acceleration.

Independent replay preserves a narrow success and explicit discrepancies

The second target computation checks every committed ID on its preceding recorded
prefix, at matching placement and KV precision, without speculative state reuse:

Declared trajectories	Matching committed IDs	Evaluated IDs
Two populated-16K continuations	256	256
Six short continuations	1,527	1,536

All cache-extension and independent-argmax checks pass. Nine short-context
discrepancies remain across four trajectories. No position is dropped and no
margin/distance exception is applied. #27
stays open; agreement on the two long trajectories is not a universal state proof
or a reclassification of historical identity failures. The bounded replay is
complete and does not block the separate stock placement experiments.

What was measured, and what remains limited

The conversation begins with 16,384 input tokens and appends code, extraction
from tool-shaped records, arithmetic, copying and a topic change. Native reused
counts exceed 16K on every continuing turn; newly evaluated counts are 39, 822,
48, 52 and 33. Every reset reports zero reuse. The larger append retains its
incremental-prefill cost. This is one scripted conversation, not autonomous tool use.

Continuing turns have a fixed 64-token cap. Four of five retained offload answers
reach it, and some are incomplete. All text is published unedited; no task-utility
score or claim that the five tasks were successfully completed is made.
The stock API also lacks an independent draft KV-reuse counter. Draft prompt
forwarding counts are explicitly derived from pinned source and target batch
accounting; proposal generation and verification catch-up remain charged work.

All 10 native processes and 1,848 inference requests complete, including 1,792
replay decisions, 48 conversation requests and 8 warmups. No native attempt fails
and no row is replaced. Peak sampled total GPU use is 14,038.395 MiB; minimum
available host memory is 12.548 GiB. Sampling is checked for coverage and gaps
but cannot establish absence of paging between observations.

The conversation configuration retains the v0.15 placement: 38 target GPU layers
including output, target threads 24/24, fully offloaded draft layers with threads
8/24, context capacity 18,432, q8_0 K/V for both models, batch/microbatch 256,
and p_min=0. The practical configuration
remains stock b10919 with the existing pinned GGUF artifacts.

Reproduction and next work

Protocol and executable preregistration
were reviewed, committed and pushed before inference. Independent review found
six receipt-validation gaps before execution; all were corrected and re-reviewed.
The final analyzer reconstructs raw HTTP/SSE, stopping, exact matrix coverage,
source identities, resource intervals and acceptance accounting.

The raw archive contains 8,775 members / 83,721,309 uncompressed bytes, including
native source snapshots, fixtures, requests, responses, timestamps, resource traces,
server logs and protocol-review receipts. Actual restoration reproduces the full
analysis byte for byte. Publication requires independent final review and the
complete suite on the final clean candidate in both available PyTorch environments.

Close bounded conversation #29
as completed while leaving the research milestone open. Next are the fixed-K16
threshold 2/4/8 comparison #30
and attention-resident / host-FFN placement #31.
The pinned runtime already provides --n-cpu-ffn; its buffer types, actual backend
assignments and total memory costs still require measurement. New drafts, learned
adaptive K and copy-overlap changes remain deferred. The ideation inventory
preserves the distinction between analytical tests, physical implementations and
unresolved hypotheses.

Detailed findings ·
Every request and discrepancy ·
Unedited outputs ·
Protocol ·
Archive inventory and restoration