v0.2.2

🚀 OpenJudge v0.2.2 Changelog

This release introduces major UI enhancements, expanded evaluation capabilities for vertical domains (Finance and Academia), and a significant architectural refactor using Executor/Strategy patterns. We have also standardized prompt templates across the board.

---

✨ New Features

🌐 Online Playground

Explore OpenJudge without writing a single line of code. Our online platform at openjudge.me/app lets you:

• Test graders interactively: Select a built-in grader, input your data, and see results instantly.
• Build custom rubrics: Use the zero-shot generator to create graders from task descriptions.
• View leaderboards: Compare model performance across evaluation benchmarks at openjudge.me/leaderboard.

📊 Domain-Specific & Advanced Evaluation

• Finance Domain: Added specialized graders for stock, event, industry, and macro-economic analysis (#117).
• Academic Paper Review: Introduced a Paper Review cookbook and a dedicated UI for systematic academic analysis (#87, #101).
• Agentic Evaluation: Added AgenticGrader, SearchCorrectnessGrader (with tool-call support), and Trajectory Accuracy graders (#82, #102).
• Multi-turn & Arena: Support for comprehensive multi-turn conversation evaluation and a new Reference Hallucination Arena (#114, #120).

🖥️ UI & User Experience

• Streamlit-based Grader UI: A new interactive interface for evaluating and managing graders (#71).
• Auto Rubric: Automatically generate grading criteria to simplify the evaluation setup (#92).
• UX Improvements: Redesigned sidebar layout, simplified theme styles, and improved the Grader Generator workflow (#103, #109).
• Analytics: Added single evaluation logging for deeper data insights (#110).

⚙️ Core Architecture

• Executor & Strategy Patterns: Implemented a more flexible backend architecture to handle diverse evaluation workflows (#97, #96).
• Experimentation: Added support for running grader evaluation experiments directly on datasets (#95).

---

🛠️ Refactoring & Optimization

• XML Prompt Standardization: Standardized prompt templates into XML tag formats for General, Agent, and Multimodal graders to improve LLM parsing reliability (#105, #108, #112).
• Reasoning-First Scoring: Reordered JSON schemas to force models to provide Reasoning before the Score, enhancing the stability and Chain-of-Thought (CoT) performance (#113).
• Pipeline Decoupling: Refactored Graders and Runners to align with the new Strategy/Executor framework (#99, #100).

---

🐞 Bug Fixes

• Environment Stability: Fixed and pinned version requirements for openai and streamlit (#79, #90).
• Paper Review Pipeline: Fixed dynamic date injection, score display issues, and added GraderError handling (#88, #91).
• UI Rendering: Resolved issues with history result rendering (#94).
• Logic Fixes: Corrected and formatted the Trajectory Accuracy grader logic (#107, #104).

---

📝 Documentation & Deployment

• Docker: Added comprehensive Docker installation guides for both OpenJudge and the training environment (#84, #111).
• Technical Docs: Updated documentation regarding Strategy, Executor, and system integration (#98).
• Project News: Updated README with the latest features including Paper Review and Auto Arena (#89).

Detailed Changelog

• Fix openai version requirement. by @weizhang25 in #79
• feat: add paper review cookbook for academic paper analysis by @XiaoBoAI in #87
• fix(paper_review): add dynamic date injection and fix score display by @XiaoBoAI in #88
• feat(ui): Add Streamlit-based grader evaluation UI by @XiaoBoAI in #71
• Fix/pin streamlit version by @XiaoBoAI in #90
• docs: add Docker installation guide for OpenJudge and training enviro… by @XieLipeng0830 in #84
• fix(paper_review): add GraderError handling in pipeline by @XiaoBoAI in #91
• docs(readme): add news for Paper Review, OpenJudge UI, and Auto Arena by @XiaoBoAI in #89
• feat: Update LLMGrader and schema by @jc200808 in #80
• fix: history result ui render by @weidankong in #94
• feat(ui): Add Auto Rubric feature for automatic grading criteria gene… by @XiaoBoAI in #92
• Feat/UI paper review by @XiaoBoAI in #101
• Feature/add trajectory accuracy grader by @helloml0326 in #102
• feature (executor): implement executor patterns by @ployts in #97
• format trajectory_accuracy_grader by @helloml0326 in #104
• feat(ui): improve Grader Generator feature with better UX by @XieLipeng0830 in #103
• refactor(graders): standardize prompt template format for common graders by @XiaoBoAI in #105
• fix trajectory_accuracy_grader by @helloml0326 in #107
• Chore/update dependencies by @XiaoBoAI in #106
• refactor(graders/agent): standardize prompts to XML tag format by @XieLipeng0830 in #108
• refactor(ui): simplify theme styles and improve sidebar layout by @XiaoBoAI in #109
• feat(ui): add single evaluation logging for analytics by @XiaoBoAI in #110
• feature(strategy): implement evaluation strategy by @ployts in #96
• feat: update docker file for judge model post training by @jc200808 in #111
• refactor(grader): refactor graders for strategy/executor by @ployts in #99
• refactor(runner): runner for grader&executor by @ployts in #100
• docs (strategy/executor/integration): update related docs for new version by @ployts in #98
• feat: add grader evaluation experiments on datasets by @jc200808 in #95
• refactor(graders): reorder JSON schema to reason before score by @XiaoBoAI in #113
• feat: add finance graders for stock/event/industry/macro analysis by @XieLipeng0830 in #117
• feat: Add multi-turn conversation graders with comprehensive evaluati… by @XieLipeng0830 in #114
• refactor(graders/multimodal): standardize prompts to XML tag format by @XiaoBoAI in #112
• Feat/ref hallucination arena by @XiaoBoAI in #120
• feat: add AgenticGrader and SearchCorrectnessGrader with tool support by @XieLipeng0830 in #82

Full Changelog: v0.2.1...v0.2.2