v0.5.5 — video input, and a 16.1% faster 256K prefill

Video input for Qwen3.8-27B, and the 256K prefill is 16.1% faster than 0.5.4.

Video

/v1/chat/completions accepts OpenAI video_url content parts, alongside the image_url support
added in 0.5.4. data: URLs only — remote fetching is refused as an SSRF primitive.

Requires ffmpeg on PATH. It is an optional runtime dependency, reported clearly if missing.

Defaults sample 2 fps, up to 32 frames, spread across the whole clip. A 6 s 448x448 clip costs
roughly 1.2K prompt tokens — frames become tokens, so the frame budget is a context budget.

Longer clips are sampled more sparsely rather than truncated. Previously a clip that exceeded the
budget lost its tail silently: the model answered about a video it had only seen the start of.

Docker image

# serve — AR + image + video, OpenAI-compatible
docker run --gpus all -p 8080:8080 -v qwen38:/models \
  ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:0.5.5

# DSpark speculative benchmark
docker run --rm --gpus all --ipc=host -v qwen38:/models \
  ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:0.5.5 bench code

Built from this tag's source by publish-docker.yml and pushed to GHCR. Weights download
themselves into the /models volume on first run. Blackwell (sm_120) only. ~1.0 GB: the image
is built on the CUDA base layer, since ldd shows the server links only libcudart — the
runtime layer would add ~3.1 GB of cuBLAS/cuSPARSE/cuFFT/cuSOLVER that nothing here references.

Provenance is attested to the image digest, the same mechanism used for the binaries:

gh attestation verify oci://ghcr.io/gittensor-ai-lab/sparkinfer-qwen38:0.5.5 \
  -R gittensor-ai-lab/sparkinfer

Concurrency

Continuous batching now decodes a whole batch in one packed forward (#975), instead of
time-slicing one sequence at a time. Aggregate throughput scales with load for the first time:

concurrent requests	before	now	gain
1	89.4 tok/s	89.0	—
2	76.5	162.3	2.1x
4	76.8	250.3	3.3x
8	77.2	343.8	4.5x
16	76.9	344.6	4.5x
32	76.5	315.2	4.1x

Distinct prompts per request so no prefix cache flatters the result, 128 tokens each, all requests
succeeding at every level. Peak is around 8-16 concurrent; single-stream decode is unchanged at
~90 tok/s.

Two supporting changes: a session's decode graph is now parked rather than destroyed on every
switch (#973), and the eval scores concurrent decode throughput directly (#978) so this cannot
silently regress.

Earlier releases also dropped requests under load; that was fixed separately by creating the
model's CUDA streams non-blocking, so a decode graph capture no longer collides with another
request's work on the legacy stream.

Long context correctness

Output above ~2,048 prompt tokens was wrong, and not reproducible under greedy decode. The
fused GQA prefill wrote its P' plane with a GN row stride while the allocation and every reader
used pld. Below 2048 tokens pad == 0, so the two agreed and every existing test passed. At or
above it, each row came back shifted, and the last row read past everything any row had written.

Fixed in #980. Same arithmetic question after a natural-prose prefix, greedy, three runs per depth:

prompt tokens	before	now
1,955	correct	correct
4,076	three different non-answers	correct x3
7,643	stable but wrong	correct x3
12,053	flaky	correct x3
24,869	391, 299, 46	correct x3
31,263	three different answers	correct x3

The eval now gates on cross-run determinism, not only within-run losslessness, so a bug of this
shape cannot ship again unnoticed.

Prefix cache

A configured prefix (SPARKINFER_SERVER_PREFIX_TOKEN_FILE) is now reused across requests instead
of being re-prefilled every time. finish_job() used to free the shared prefix session outright,
so prefix_cached_len() came back 0 and the whole prefix was recomputed on each call — the cache
only ever avoided re-prefill within one request.

Two things were needed, and only the first is obvious: keep the prefix's KV blocks
(truncate_blocks() rather than free()), and restore the Gated-DeltaNet recurrent state.
48 of Qwen3.8's 64 layers carry state that is not paged and that decoding mutates in place, so
keeping the KV alone would have left them holding the previous request's history.

32,022-token shared prefix, wall clock:

request	before	now
1 (cold)	5.88 s	5.87 s
2	5.85 s	0.45 s
3	5.72 s	0.31 s

~14x on every request after the first. Sequential reuse only — the prefix_exclusive gate is
unchanged, so this does not engage under concurrency.

Tool calling

$schema and $comment are accepted in tool parameter schemas (#972). Clients built on
@ai-sdk/openai-compatible — Kilo among them — put $schema on every tool schema, and it was
rejected, so tool calling returned 400 on every request from those clients. Both are pure
annotations a validator must ignore; they are still type-checked as strings.

Structural $ keywords ($ref, $defs, $id) are still refused on purpose: silently ignoring a
$ref would validate the model's arguments against nothing. A schema that carries $defs — zod
emits it for reused or recursive sub-schemas — still returns 400.

Prefill

Two bit-identical optimisations to the scored 256K pass. Decode unchanged at 94.4 tok/s.

	prefill tok/s @ 262144
0.5.4	3,725
0.5.5	4,326 (+16.1%)

Benchmarks

Both tables are unchanged from 0.5.4 and were not re-measured for this release.

Versus SGLang — same prompts, same box, 4K / 128 tokens.

workload	SparkInfer	SGLang	delta
chat	167.92 tok/s	114.56	+46.6%
code	171.89 tok/s	170.32	+0.9%
math	249.08 tok/s	225.09	+10.7%
JSON	202.88 tok/s	202.33	+0.3%
repetition	406.85 tok/s	398.61	+2.1%

DSpark speculative decode — RTX 5090, greedy, batch 1, 128-token output, byte-identical to AR.

	4K	16K	32K
mean speedup over AR	4.01x	2.97x	2.63x

AR reference: 91.0 tok/s at 4K, 86.4 at 16K, 81.3 at 32K.

The two tables use different prompt corpora and are not comparable to each other — the same
engine measures 167.9 on one and 306.0 on the other. No SGLang claim is made at 16K or 32K.

Also

• tool_choice=required now verifies a tool call was actually emitted, instead of accepting prose.
• Interleaved MRoPE for multimodal positions (SPARKINFER_MROPE=0 to disable).
• The server reports the model it loaded and advertises image input when a vision tower is present.
• server/scripts/diagnose_concurrency.py — one-shot 6/8/10/12-stream diagnostic report.
• ttft_ms, generation_ms and decode_tps are engine telemetry, not billing or reward figures.
• SPARKINFER_MAX_QUEUE_DEPTH=0 means unlimited admission; set a finite cap to reject on overload.