Skip to content

v0.2.1

Choose a tag to compare

View all tags
@helloml0326 helloml0326 released this 21 Jan 06:34
· 96 commits to main since this release
v0.2.1
595f1fd
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Changelog

Integrations & Ecosystem

  • VERL Integration: Added an integration guide for VERL with async reward evaluation support.

  • Observability: Added documentation and cookbooks for Langfuse and LangSmith integrations.

Grader Improvements

  • Standardized Scoring: Adjusted the score range for multimodal graders from [0, 1] to a 1-5 scale for better granularity.

  • Tool Call Evaluation:

    • Added ToolCallPrecisionRecallMatchGrader.

    • Renamed ToolCallSequenceMatchGrader to ToolCallStepSequenceMatchGrader and added metric_type parameters.

  • Terminology Alignment: Renamed "Reward Model" to "Judge Model" across documentation and code to better reflect its role in the evaluation ecosystem.

  • Refining Logic: Standardized parameter naming, response parsing, and improved streaming support across various graders.

Evaluation Pipeline

  • Zero-shot Evaluation Pipeline: Added a comprehensive zero-shot evaluation workflow, including win rate chart generators, report generators, and rerun-judge capabilities.

  • GRPO Enhancements: Improved code style for GRPO, added a dedicated report generator, and updated training documentation for judge models.

  • Rubric System: Introduced and enhanced Simple Rubric and Iterative Rubric generation and evaluation modules.

  • Dataset Updates: Updated ChatRLDataset class and associated logic to better support reinforcement learning workflows.

What's Changed

  • Feature/zero shot evaluation by @XiaoBoAI in #28
  • docs: add DingTalk community group QR code to README by @helloml0326 in #30
  • docs: add docs/integration/langfuse.md by @helloml0326 in #31
  • refactor(grpo): improve code style and add report generator by @XiaoBoAI in #37
  • feat(integration): add LangSmith integration cookbook and documentation by @ployts in #33
  • Minor code refactoring, including None checks and argument name changes. by @weizhang25 in #34
  • feat(template): add GitHub PR template and issue templates by @ployts in #36
  • Docs/sample reports by @XiaoBoAI in #38
  • Rename grader validator file name from base.py to grader_validator.py by @weizhang25 in #41
  • A common util method of formatting history for agent graders. by @weizhang25 in #42
  • feat: update agent graders by @jc200808 in #35
  • docs: add Simple Rubric documentation and rename to Generate Rubrics … by @XieLipeng0830 in #45
  • feat: add report generator and update zero-shot evaluation pipeline by @XiaoBoAI in #32
  • docs(building_graders): add training reward models guide and update integrations by @XiaoBoAI in #46
  • chore(test): remove redundant multimodal graders syntax test by @XiaoBoAI in #43
  • feat: update example code and template processing in agent and common graders by @jc200808 in #49
  • Use compiled regex objects instead of raw pattern strings. by @weizhang25 in #48
  • Docs/rename reward to judge model by @XiaoBoAI in #47
  • feat(security): add pre-commit hooks for secret detection by @XiaoBoAI in #50
  • refactor(graders): standardize parameter naming and response parsing by @ployts in #44
  • docs: update index page with new features by @helloml0326 in #51
  • fix(graders): fix typo in code_execution filename and imports by @XiaoBoAI in #54
  • Set the name of a customized llm grader. by @weizhang25 in #52
  • refactor(graders,models): cleanup and improve code quality by @XiaoBoAI in #55
  • Add OpenJudge integration guide for VERL with async reward evaluation. by @chr6192 in #53
  • refactor(graders): fix deprecated parameters and improve input validation by @XiaoBoAI in #56
  • feat(zero_shot): add win rate chart generator by @XiaoBoAI in #57
  • A utility function that collects all grader information. by @weizhang25 in #58
  • fix(graders): modify threshold value by @weidankong in #59
  • feat: update more graders including multimodal graders by @jc200808 in #60
  • refactor(analyzer): update consistency analyzer and agent grader tests by @ployts in #61
  • refactor(graders): improve parameter validation and streaming support by @XiaoBoAI in #62
  • feat(grader): add metric_type parameter to ToolCallSequenceMatchGrader by @helloml0326 in #64
  • fix: fix a typo in a comment by @Wangzy455 in #66
  • feat: update ChatRLDataset class and judge model grpo training document by @jc200808 in #67
  • Update get_all_grader_info logic. by @weizhang25 in #65
  • feat(zero_shot): enhance evaluation pipeline with rerun-judge and cha… by @XiaoBoAI in #63
  • chore: migrate links from modelscope to agentscope-ai org by @XieLipeng0830 in #73
  • refactor(multimodal): change score range from [0,1] to 1-5 scale by @XiaoBoAI in #75
  • feat(iterative_rubric): enhance rubric generation and evaluation by @XieLipeng0830 in #72
  • Update grader info util. by @weizhang25 in #74
  • docs: improve README with comprehensive examples and update simple ru… by @helloml0326 in #76
  • Docs/update verl integration status by @helloml0326 in #77
  • feat: update base/openai chat model module by @jc200808 in #70
  • Feature/add ToolCallPrecisionRecallMatchGrader & rename: ToolCallSequenceMatchGrader ->ToolCallStepSequenceMatchGrader by @helloml0326 in #68

New Contributors

Full Changelog: https://github.com/agentscope-ai/OpenJudge/commits/v0.2.1

Contributors

chr6192, helloml0326, and 7 other contributors
Assets 2
Loading