Run models in parallel during benchmark #53
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR enhances the guardrail evaluation tool to run model benchmarks in parallel rather than sequentially, significantly improving performance when testing multiple models. The changes introduce concurrency controls and chunking capabilities for better resource management during benchmarking.
Key Changes:
- Models now run concurrently during benchmarking, controlled by a semaphore with configurable parallelism limits
- Added optional sample chunking to break up large benchmark runs into smaller pieces
- Enhanced structured logging throughout the benchmark process with timing information
- Added comprehensive unit tests for the new parallel execution and chunking functionality
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| src/guardrails/evals/guardrail_evals.py | Implemented parallel model execution with asyncio.Semaphore, added chunk_samples method, enhanced logging with structured events and timing, added CLI arguments for controlling parallelism |
| tests/unit/evals/test_guardrail_evals.py | New test file with unit tests covering parallel model limit determination and sample chunking functionality |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@codex review |
|
Codex Review: Didn't find any major issues. Breezy! ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
Small change to the eval tool.