Skip to content

v0.1.1

Choose a tag to compare

@minhnguyent546 minhnguyent546 released this 07 Jul 16:01
v0.1.1
111eba6

Features:

  • Support specifying a custom self-hosted judge model via the hosted_vllm/ prefix (hosted_vllm/deepseek-v4-pro, hosted_vllm/deepseek-v4-flash), backed by a new VIREX_BENCH_JUDGE_BASE_URL env var that points the LLM-as-a-judge metric at an OpenAI-compatible serving endpoint (d940208)

Scripts:

  • Qwen3-4BQwen3-4B-Instruct-2507 in the SGLang serving script (f044801)
  • Add eval scripts for large models: Gemma 4 (26b-a4b-it, 31b-it), Qwen3.5 (9b, 27b, 35b-a3b, 122b-a10b-fp8), Qwen3.6 (27b, 35b-a3b) (af994c1)
  • Add serving script for phi-4-mini-reasoning (vLLM + SGLang) (5ae9969)
  • Add vLLM serving scripts for NVIDIA-Nemotron-3-Nano-4B-BF16, NVIDIA-Nemotron-3-Super-120B-A12B-FP8, and NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 (b5a8931, b09a304, d9c2662)
  • Add eval scripts for the Nemotron-3 family and phi-4-mini-reasoning (e17b525)
  • Add SGLang serving script for Qwen3.5-122b-a10b-fp8 and update the Nemotron-3-Super-120B vLLM script (6f9e158)

Fixes:

  • Set EXTRA_PARAMS in the large-model eval scripts (Gemma 4, Qwen3.5/3.6 variants) (eef0bc7)
  • Fall back an empty-string API key to <empty> across the run CLI, judge LM builder, and model backends, since some providers reject an empty API key (1e4938b)

Docs:

  • Correct the README usage guide for VIREX_BENCH_JUDGE_MODEL (41700d6)