Dev/steven/parallel eval #43
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR implements a benchmark mode for the evaluation tool to achieve parity with the Python version. The changes introduce parallel model benchmarking, latency testing, advanced metrics calculation, and visualization capabilities.
- Adds benchmark mode with parallel model evaluation and chunking support
- Implements advanced metrics (ROC AUC, precision at recall thresholds) for model comparison
- Introduces latency testing infrastructure with configurable iteration counts
Reviewed Changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/evals/guardrail-evals.ts | Main evaluation runner with new benchmark mode, parallel model evaluation, and multi-stage support |
| src/evals/core/benchmark-calculator.ts | Advanced metrics calculator for ROC AUC, precision-recall curves, and FPR-based recall |
| src/evals/core/benchmark-reporter.ts | Benchmark results reporter with structured output and summary tables |
| src/evals/core/latency-tester.ts | Latency testing module for measuring guardrail performance |
| src/evals/core/visualizer.ts | Stub implementation for benchmark visualization (placeholder) |
| src/evals/core/index.ts | Export updates for new benchmark modules |
| src/evals/core/async-engine.ts | Minor refactoring to use totalSamples variable |
| src/cli.ts | CLI argument parsing for benchmark mode, stages, models, and API configuration |
| src/tests/unit/evals/guardrail-evals.test.ts | Unit tests for chunking and parallel model limit logic |
| docs/evals.md | Documentation updates for new benchmark features |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Updating eval tool