v0.1.1
Features:
- Support specifying a custom self-hosted judge model via the
hosted_vllm/prefix (hosted_vllm/deepseek-v4-pro,hosted_vllm/deepseek-v4-flash), backed by a newVIREX_BENCH_JUDGE_BASE_URLenv var that points the LLM-as-a-judge metric at an OpenAI-compatible serving endpoint (d940208)
Scripts:
Qwen3-4B→Qwen3-4B-Instruct-2507in the SGLang serving script (f044801)- Add eval scripts for large models: Gemma 4 (26b-a4b-it, 31b-it), Qwen3.5 (9b, 27b, 35b-a3b, 122b-a10b-fp8), Qwen3.6 (27b, 35b-a3b) (af994c1)
- Add serving script for
phi-4-mini-reasoning(vLLM + SGLang) (5ae9969) - Add vLLM serving scripts for
NVIDIA-Nemotron-3-Nano-4B-BF16,NVIDIA-Nemotron-3-Super-120B-A12B-FP8, andNVIDIA-Nemotron-3-Nano-30B-A3B-BF16(b5a8931, b09a304, d9c2662) - Add eval scripts for the Nemotron-3 family and
phi-4-mini-reasoning(e17b525) - Add SGLang serving script for
Qwen3.5-122b-a10b-fp8and update the Nemotron-3-Super-120B vLLM script (6f9e158)
Fixes:
- Set
EXTRA_PARAMSin the large-model eval scripts (Gemma 4, Qwen3.5/3.6 variants) (eef0bc7) - Fall back an empty-string API key to
<empty>across the run CLI, judge LM builder, and model backends, since some providers reject an empty API key (1e4938b)
Docs:
- Correct the README usage guide for
VIREX_BENCH_JUDGE_MODEL(41700d6)