Skip to content

Evalution v0.0.6

Choose a tag to compare

View all tags
@Qubitium Qubitium released this 16 Apr 15:44
· 40 commits to main since this release
v0.0.6
24e393b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Notables

  • Added runnable benchmark implementations for hle, supergpqa, hmmt_feb25, hmmt_nov25, hmmt_feb26, imoanswerbench, and livecodebench_v6.
  • Registered capability-gated placeholders for swe_bench_verified, swe_bench_multilingual, swe_bench_pro, terminal_bench_2, claw_eval_avg, claw_eval_pass3, skillsbench_avg5, qwenclawbench,
    nl2repo, qwenwebbench, tau3_bench, vita_bench, deepplanning, tool_decathlon, mcpmark, mcp_atlas, and widesearch, with clear runtime-capability errors instead of misleading partial implementations.
  • Exported the new suites through evalution.benchmarks and added integration metadata/baselines in tests/models_support.py.
  • Added unit coverage plus standalone Llama 3.2 1B Instruct regression tests for the new runnable suites.
  • Hardened math answer extraction to handle boxed answers, explicit final-answer lines, and inline math spans more reliably, using compiled pcre patterns.
  • Added an optional apply_chat_template mode for HLE while keeping the default benchmark-faithful prompt path unchanged.

What's Changed

Full Changelog: v0.0.5...v0.0.6

Contributors

Qubitium
Assets 2
Loading