Skip to content

v0.7.4

Latest
Latest

Choose a tag to compare

View all tags
@pufanyi pufanyi released this 30 Oct 13:51
· 3 commits to main since this release
v0.7.4
c144e90
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

What's Changed

  • docs: update localized README hero images by @pufanyi in #1484
  • docs: fix hero image rendering on PyPI by @pufanyi in #1485
  • chore: update PyPI homepage URL by @pufanyi in #1486
  • fix(video_llava): pass num_frm to read_video by keyword by @Anai-Guo in #1487
  • fix(vllm): avoid eager Decord loading by @pufanyi in #1480
  • feat: add LongVideo-Reason long-video reasoning benchmark by @ngquangtrung57 in #1488
  • fix: report aggregate MVBench accuracy by @pufanyi in #1489
  • fix(distributed): call destroy_process_group() on exit to prevent NCC… by @mandanajignesh-byte in #1391
  • fix(oryx): make the video call sites match what load_video/read_video return by @Anai-Guo in #1493
  • fix(llava_vid): call read_video with its actual signature on the pyav path by @Anai-Guo in #1492
  • fix: Qwen3-VL pixel budgets to use the 32×32 visual-token grid by @gertelrina in #1490
  • fix(mdpbench): pass line_type when match_gt2pred_no_split calls get_gt_pred_lines by @Anai-Guo in #1500
  • style: fix Qwen3-VL lint failure by @pufanyi in #1501
  • fix(qwen3_vl): preserve the 256-2048 visual-token budget by @pufanyi in #1502
  • feat: vLLM-Omni and sglang-diffusion for video generation models by @pufanyi in #1314
  • style: fix lint for video generation backends by @pufanyi in #1503
  • feat: add HERBench benchmark (CVPR 2026, multi-evidence video QA) by @GabrieleSerussi in #1504
  • feat: add normalized physics benchmarks by @Luodian in #1272
  • ci: (PR 1/2) add the hermetic CPU contract suite by @Luodian in #1440
  • test: (PR 2/2) remove mirrored request-layout checks by @Luodian in #1451
  • security(tui): (collector) land structured execution and origin guards by @Luodian in #1445
  • fix(evaluator): resolve sample limits independently per task by @Luodian in #1441
  • fix(cache): validate recovered responses and repair invalid entries by @Luodian in #1513
  • fix: change decord to lazy import by @microslaw in #1517
  • feat: add MET-Bench entity tracking tasks by @vanyacohen in #1518
  • fix (vllm_generate): forward trust_remote_code to AutoProcessor by @jbyczkow in #1531
  • fix(vllm_generate): fix video metadata handling for zai-org/GLM-4.1V-9B-Thinking by @jbyczkow in #1535
  • chore: upgrade PyAV to 17.1 by @pufanyi in #1536
  • fix: bound memory during PyAV packet sampling by @pentaoa in #1520
  • chore: bump package version to 0.7.3 by @pufanyi in #1537
  • chore: enable Ruff builtin-shadowing and async checks by @pufanyi in #1538
  • feat(mmmu): support local evaluation of the public test split by @MingheShen in #1515
  • ci: require end-to-end evidence for new integrations by @pufanyi in #1540
  • feat(gemma4): add native audio eval for E2B, E4B, and 12B by @devledas in #1539
  • fix(pointbench): draw the reference point of steerable questions by @superAIyah in #1532
  • style: format Gemma 4 model by @pufanyi in #1542
  • ci: enforce E2E evidence for existing evaluation code by @pufanyi in #1547
  • fix(mmmu_pro): parse vision answers and read the explicit 'Answer: X'… by @superAIyah in #1545
  • feat(mmstar): add mmstar_hybrid, a per-question answer-format protocol by @superAIyah in #1522
  • fix(mmmu_pro): use the official direct.vision prompt for mmmu_pro_vision by @superAIyah in #1549
  • feat: add E3VQA benchmark by @Leeinsu1 in #1551
  • Set padding_side="left" for batched generation in two model wrappers by @aviadarn in #1550
  • fix(countbenchqa): drop caption from prompt to stop leaking the answer by @danny0122 in #1556
  • feat: add MVH-Bench evaluation tasks by @Leeinsu1 in #1559
  • fix(reasoning): score mcq only when the gold is a bare option letter by @longzhenren in #1553
  • Add TimeBlind video benchmark task by @Baiqi-Li in #1560
  • feat: add Video-Index benchmark by @Espere-1119-Song in #1561
  • docs: define a safe modular evaluation pipeline by @Luodian in #1439
  • fix(cache): rebuild live requests from cached contexts by @Luodian in #1514
  • fix: recognize lowercase MCQ answers and stop prose false positives in mcq_extract by @longzhenren in #1554
  • fix: parse shared MCQ answers with scoped evidence by @pufanyi in #1564
  • test: add benchmark-specific adversarial MCQ fixtures by @pufanyi in #1565
  • feat: add DocAtlas-Bench benchmark by @ahmedheakl in #1562
  • fix: support Qwen3-VL likelihood evaluation on MMLU by @pufanyi in #1566
  • Fix typos in README and docs by @Jacky1128 in #1567
  • fix(groundingme): score unparsed responses as wrong and fix resized-p… by @superAIyah in #1570
  • feat: add Spatial Scene Bench tasks by @selfishout in #1574
  • fix: composite transparency onto white before RGB conversion by @Imad2702 in #1573
  • fix: skip "nan" hints in MMBench-CN prompts by @surgan12 in #1578
  • feat: add SPaRC visual pathfinding tasks by @lkaesberg in #1579
  • fix(charxiv): select reasoning instructions and rubrics by answer type by @surgan12 in #1583
  • fix(mmmu): keep question text when an image is referenced twice by @mottopanikeiku in #1589
  • feat: add VisualOverload benchmark with private-answer submissions by @paulgavrikov in #1582
  • feat: add OmniCompass VL and AV evaluation by @xuyang-liu16 in #1584
  • fix: accept task YAML files passed to --tasks by @mottopanikeiku in #1588
  • fix: compute POPE yes ratio from predictions by @mottopanikeiku in #1592
  • docs: fix relative links with wrong directory depth by @pratikgx in #1600
  • fix(ci): move task input dataset cache off tmpfs by @pufanyi in #1602
  • sparc_visual: state the negative polyshape rule as the puzzles use it by @lkaesberg in #1596
  • fix(mathvision): restore math names when evaluating latex2sympy output by @mottopanikeiku in #1595
  • fix(openai): disable SDK retries to avoid nested retry loops by @sangk-kim in #1593
  • fix(babyvision): don't count judge failures as wrong answers by @FazeelUsmani in #1597
  • fix(mmstar): score categories and average over questions like the official scorer by @mottopanikeiku in #1603
  • feat(tasks): add AI2D task with per-question images by @jiahaozheng406 in #1590
  • Massive fixing of registed → registered by @Jacky1128 in #1568
  • Fix typos in model comments and example scripts by @Jacky1128 in #1569
  • fix: preserve masks for pointing task scoring by @oscarqjh in #1611
  • Feat/mavis benchmark by @seokwon99 in #1607
  • fix: parse bold MCQ labels and Chinese colon answers by @mengchen-atx in #1609
  • feat(mathvision): add mathvision_relaxed_eval with the four extraction fixes by @feiiiiii5 in #1552
  • feat(countqa): add CountQA benchmark for counting in real-world photos by @phantom-vs in #1612
  • feat(humanref): add HumanRef multi-person referring benchmark by @phantom-vs in #1613
  • refactor(models): make ModelRegistryV2 authoritative by @Luodian in #1450
  • fix(eval): abort all ranks on per-rank exception instead of deadlocking by @Luodian in #1304
  • perf: add evaluator phase recording and reproducible benchmark harness by @Luodian in #1460
  • fix(api/task): guard empty results list in process_results by @Luodian in #1305
  • feat(agentic): run ViZDoom and MiniGrid through the existing agentic loop by @Luodian in #1418
  • ci: align lint and contributor setup with Python 3.10 by @Luodian in #1615
  • feat: add TempCore temporally sensitive video QA subsets by @HJ-Ok in #1617
  • fix(cache): publish local SQLite snapshots safely to NFS by @pufanyi in #1618
  • fix(models): preserve per-request chat inference by @WenzheWang in #1619
  • feat(mavis): add the Multi-, Text- and Image-RAG settings by @seokwon99 in #1620
  • feat: add Qwen3.6 model support by @devledas in #1585
  • docs: fix quickstart and guide navigation links by @Luodian in #1616
  • chore: bump version to 0.7.4 by @pufanyi in #1621

New Contributors

Full Changelog: v0.7.3...v0.7.4

Contributors

pratikgx, FazeelUsmani, and 38 other contributors
Assets 2
Loading