Skip to content

v0.7.3

Latest
Latest

Choose a tag to compare

View all tags
@pufanyi pufanyi released this 28 Aug 14:33
v0.7.3
ac86d61
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

  • fix(visualwebbench): allow anonymous dataset load by @ts-kim in #1370
  • fix(grounding/rec): guard empty metric bucket against ZeroDivisionError by @ts-kim in #1371
  • feat: add CC-OCR benchmark by @kcz358 in #1372
  • feat: add refcoco/refcoco+/refcocog grounding tasks by @kcz358 in #1373
  • feat: add ODinW-13 open-vocabulary detection task by @kcz358 in #1374
  • feat(odinw13): add COCO mAP metrics by @kcz358 in #1375
  • feat: add SUNRGBD 3D object detection task by @kcz358 in #1376
  • fix(llm_judge): OpenAI base URL, max_completion_tokens for reasoning models, PIL image support by @Luodian in #1381
  • feat(gemini): migrate to google-genai SDK, keep gemini_api alias by @Luodian in #1382
  • feat(entrypoints): job scheduler improvements for the HTTP eval server by @Luodian in #1383
  • fix: core evaluator/cache/utils correctness fixes by @Luodian in #1378
  • feat(tasks): add IntPhys2 and VANTAGE-Bench VQA by @Luodian in #1387
  • feat(loggers): add SwanLab logger as a wandb alternative (--swanlab_args) by @Luodian in #1385
  • feat: add verifiers module - pluggable per-sample verification pipeline by @Luodian in #1379
  • feat: auto-strip thinking blocks from generations before scoring by @Luodian in #1380
  • fix(tasks): utils improvements across 7 benchmarks by @Luodian in #1386
  • refactor: remove Lance-based data paths — HF/local data hosting only by @Luodian in #1384
  • feat(models): add diffusers-based world-model generation backends (Wan 2.1/2.2, Cosmos, LTX-Video, MAGI-1, fal.ai) by @Luodian in #1389
  • fix(packaging): include extensionless task template files in pip wheel by @mandanajignesh-byte in #1390
  • feat(tasks): add SLAKE bilingual medical VQA benchmark by @hiajoo in #1392
  • Add MMOU and RVOS tracking benchmarks by @kcz358 in #1394
  • fix(rvos, mmou): parse truncated tracks + switch MMOU to self-contained repo by @kcz358 in #1395
  • refactor(mmou, rvos): write submissions under <output_path>/submissions/ by @kcz358 in #1396
  • feat: add MultihopSpatial task by @youngwanLEE in #1402
  • feat(task): add SIS-Bench evaluation by @choucisan in #1401
  • fix: raise RoboSpatial context generation limit by @lx88882222 in #1400
  • fix: key RoboSpatial mask cache by content by @lx88882222 in #1399
  • feat: add CG-Bench and CG-AV-Counting tasks by @jscslld in #1405
  • Add LFM2.5-VL model wrapper by @Saad1926Q in #1411
  • feat(tasks): add MDPBench benchmark by @Delores-Lin in #1407
  • Add Qwen-native JSON coordinate variants for pointing tasks by @lx88882222 in #1409
  • feat: add Kaleidoscope multilingual exam benchmark by @ShayekhBinIslam in #1412
  • feat: add SlideVQA benchmark by @Saad1926Q in #1413
  • feat(videoevalpro): add regex-scored MCQ task by @kcz358 in #1414
  • fix: normalize fullwidth ASCII in OCRBench scoring by @turturturturtur in #1415
  • chore: migrate dataset paths to new orgs by @kcz358 in #1416
  • feat: add Multi-Crit benchmark by @tyxiong23 in #1417
  • Codex/physical conflict atomic v1 by @bht1027 in #1422
  • fix: build SGLang sampling params per request instead of per batch (#1397) by @Anai-Guo in #1423
  • feat: add Aero Realtime vLLM-Omni adapter by @kcz358 in #1424
  • [Model] Add Neo-OV model support and SI benchmark configs by @oscarqjh in #1403
  • fix(tasks): handle zero-correct VLMSAreBlind results by @FatimaJahara in #1437
  • fix(blink): restrict answer extraction to valid choices and parse verbose responses by @akawincent in #1435
  • [Model] Add Mistral3-VL simple model by @thanksmumu in #1425
  • fix(openai): preserve per-request sampling parameters by @turturturturtur in #1426
  • feat(tasks): add owner-aligned MicroVQA benchmark by @turturturturtur in #1427
  • fix(vllm): preserve per-request sampling parameters by @turturturturtur in #1428
  • feat(tasks): add C4 Bench by @sci-m-wang in #1429
  • [Model] Add VQToken integration for LLaVA-OneVision by @Hai-chao-Zhang in #1430
  • fix: carry full task ctx in auto doc_to_messages for text-only tasks (#1319) by @lorenzozanee in #1431
  • fix: cap nframes to actual video frame count in qwen3_vl by @Renard77 in #1433
  • feat: add VCRBench, MMR-V and VRBench video reasoning benchmarks by @ngquangtrung57 in #1438
  • fix(mmmu): preserve full candidate list for open-ended eval_open scoring by @ClaireXi99 in #1471
  • docs: optimize README hero image by @pufanyi in #1478
  • feat: add gemma4 model support by @EliYuan30 in #1474
  • fix: handle verbose answers in videomme_v2 evaluation by @EliYuan30 in #1475
  • Fix missing Task import for type annotation in evaluator by @luv-oct22 in #1291
  • fix: add video evaluation support to gemma3 by @EliYuan30 in #1473
  • fix(chat-vllm): add is_qwen3_vl flag to route video frames through timestamped encoding by @feiiiiii5 in #1472
  • feat: add VBench video generation evaluation benchmark by @Luodian in #1271
  • chore: adopt Ruff formatter and default lint rules by @pufanyi in #1479
  • perf: parallelize document result processing by @pufanyi in #1477
  • ci: use uv across Python workflows by @pufanyi in #1482

New Contributors

Full Changelog: v0.7.2...v0.7.3

Contributors

youngwanLEE, jscslld, and 28 other contributors
Assets 2
Loading