giq 0.5.1 — production fixes and faster NVFP4 recipes

giq 0.5.1 fixes three problems found on the first 0.5.0 production deploy, and tunes the NVFP4 Qwen3.8-27B recipes for batch extraction on the strength of a benchmark on an RTX 5090. No breaking changes; upgrade in place.

Fixes

• Non-streaming chat answers were rebuilt by giq, and lost things. A /v1/chat/completions request without tools, images or streaming kept only the last system and user message, and giq wrote the answer itself with finish_reason: "stop". So:

  • an answer cut off by max_tokens while the model was still thinking came back empty and looked finished;
  • the earlier turns of a conversation never reached the model;
  • reasoning_content was dropped.

  Every non-streaming chat now goes to the engine whole, as streaming and tool calls already did, and the engine's answer comes back as written: "length" when the budget ran out. To keep thinking from eating the answer, cap it with reasoning_budget_tokens (llama.cpp), or turn it off for extraction with chat_template_kwargs: {"enable_thinking": false}.

• /status reported 0 GB in use by giq on a card running a vllm instance. vllm's API server holds no VRAM; its engine-core child does. Attribution now counts the processes a spawned engine starts.

• /status reported version 0.4.0 on a 0.5.0 deploy. The package carried its own copy of the version string, which the release bump missed. It now reads the installed version, and a test holds it to pyproject.toml. Run make sync after checking out a new version, as the update steps say.

Faster NVFP4 recipes, measured

Two recipes over the same nvidia/Qwen3.8-27B-NVFP4 checkpoint, each tuned for its job:

• qwen3.8-27b-nvfp4 (batch): schedules 16 requests at once instead of 32. 16 already saturate the card; more only wait in the queue.
• qwen3.8-27b-nvfp4-chat (one fast answer): drafts 3 tokens with the model's own multi-token-prediction head instead of 2, with the KV budget raised to the 5.5 GiB that three drafts need for a full 128k request.

Benchmark report

RTX 5090 (32 GB, power limit 500 W), vllm 0.30. Batch extraction: a unique ~3.1k-token document per request (real text from this repository), a shared extraction prompt, ~490 tokens of JSON out under a JSON schema, thinking off, temperature 0. Every one of the ~1,700 outputs was valid JSON with finish_reason: stop.

Batch throughput — the card is compute-bound at about 4,200 tokens/s on this model:

Configuration	Documents/min	Input tok/s	Output tok/s	Slowest 5%
16 parallel, no speculation	71	3,640	580	27 s
32 parallel (0.5.0's recipe)	68	3,500	550	139 s
64 parallel	65–71	3,350–3,670	530–580	~140 s
64 parallel, 16k tokens per step	ran out of memory under load
16 parallel, MTP 1 draft token	47	2,400	380	112 s
16 parallel, MTP 2 draft tokens	66	3,400	540	70 s

A single request — speculation is close to free here, and JSON drafts well:

Draft tokens	0	1	2	3
vllm, NVFP4 (tok/s)	69	108	138	162
llama.cpp, Q6_K --spec-type draft-mtp (tok/s)	52	72	82	89
vllm, seconds per document	7.5	4.8	3.7	3.1

The NVFP4 checkpoint and the Q6_K GGUF are the same size (20.4 GiB), so one request reads the same bytes per token on both: the gap is the engine.

Power limit (the 5090 allows 400–575 W):

Limit	Batch documents/min	Drawn	Documents per kWh	Single request tok/s
575 W	72	481 W	8,990	162
500 W	71	465 W	9,130	164
450 W	69	442 W	9,460	164
400 W	67	396 W	10,180	163

For batch, 400 W gives 13% more documents per kWh for 7% fewer per minute. A single request never draws more than ~410 W.

What else the benchmark found:

• Prepare vllm after every upgrade. On an empty cache, vllm 0.30's first start compiled kernels for 14 minutes (819 s), past giq's 600 s start budget. Run giq prepare vllm --recipe <name> after each vllm upgrade.
• A larger step budget doesn't help. max_num_batched_tokens bought nothing on this workload. With a KV size given in bytes, vllm skips its memory profiling, so a step budget too big for the card fails under load rather than at start.
• Prefix caching needs long shared prompts. Qwen3.8 is a hybrid model, and vllm caches its prefixes in blocks of 1,568 tokens. A shared prompt shorter than that gets no cache hits; this one was ~400 tokens.
• The interactive recipe is tight on a 5090. It declares 29.7 GB and passes giq's gate on a 32 GB card with 0.1 GB to spare. On a card that also drives a display it won't load. On larger cards (RTX PRO 6000), replace both files in your recipes directory with a larger kv_cache_memory, and measure there: compute per card is similar, so the 16-parallel saturation point likely holds.

Full changelog: v0.5.0...v0.5.1