Skip to content

OpenJudge v0.2.5

Latest

Choose a tag to compare

@helloml0326 helloml0326 released this 10 Nov 08:06
5d0d9a4

OpenJudge v0.2.5

This release brings the current OpenJudge main branch to PyPI, focusing on grading correctness, reliable model requests, cancellation cleanup, and safer Auto Arena configuration handling. It also includes provider, harness, and skill improvements merged since the previous PyPI release.

Grading correctness

  • Distinguish JSON booleans from numbers and missing fields from explicit null; correctly match unordered JSON records (#205, #207, #218).
  • Reject NaN and Infinity literals during JSON validation while preserving valid string values (#226).
  • Reject non-object tool-call payloads with an invalid-format score instead of raising an exception (#215).
  • Correct scientific-notation parsing and prevent punctuation/stop-word-only F1 and Jaccard inputs from receiving perfect scores (#202, #216).
  • Fix degenerate empty-string matching cases (#200).
  • Honor per-call SimilarityGrader options over constructor options (#220).
  • Restore multimodal evaluation instructions, rubrics, context, and queries in image coherence, image helpfulness, and text-to-image requests (#224).

Reliability and model compatibility

  • Drain cancelled runner work and sibling evaluation tasks before propagating cancellation/errors (#208).
  • Validate positive concurrency and restore consistency-analyzer call compatibility (#203, #204).
  • Report analyzer coverage, reject misaligned inputs, and handle zero-variance correlation inputs (#198, #191).
  • Preserve reusable LLMGrader configurations and restore string-to-enum conversion (#222).
  • Use compatible JSON-object responses for DeepSeek (#213).
  • Format Qwen Omni audio once without mutating caller-owned messages (#228).
  • Honor per-call OpenAI streaming overrides consistently (#229).
  • Include LiteLLM support and request-handling improvements, QwenVL timeout support, and MiniMax updates (#185, #193, #196, #197).

Auto Arena credentials

  • Remove credential values from saved UI configurations, presets, and exported YAML while preserving in-memory credentials (#212).
  • Supply credentials in memory when resuming sanitized runs. Automatically reused UI credentials are bound to endpoint URLs; legacy credentials are preserved.

Evaluation workflows

  • Include the grader evaluation harness, feedback-schema improvements, harness-based agentic grading, and reorganized evaluation skill suites (#179, #182, #187, #188, #189).

Upgrade notes

pip install --upgrade py-openjudge==0.2.5
  • Several fixes intentionally change scores for previously misgraded inputs. Record the package version when comparing evaluation results; rerun affected evaluations when consistent scoring is required.
  • Resuming newly saved Auto Arena runs requires credentials for the saved endpoints. Earlier plaintext files are not retroactively sanitized.
  • Pending confidence-calibration (#211) and Markdown-fence parsing (#209) changes are not included.
  • Existing migration requirements from legacy rm-gallery remain unchanged.

Thanks to all contributors who supplied reports, reproductions, fixes, and regression tests.