docs(benchmarks): Add production and Kimi results - #517
Merged
Conversation
Record standardized production, Kimi K2.6, and Kimi K3 effort runs. Track exact production model lanes so published comparisons do not inherit stale labels. Co-Authored-By: GPT-5.6 Sol <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Looks good to me. The production profile matches the pinned getsentry/.github warden.toml, the new validator correctly requires a stable row for both model lanes, and the published numbers line up with the committed result files (including the July vs current overlap and the Kimi turn/token figures). Nice call documenting the aux-merge drop so 37/86 reads as an end-to-end system score.
gricha
marked this pull request as ready for review
August 22, 2026 15:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publish four standardized security-review corpus results for the current
production model stack, Kimi K2.6, and Kimi K3 at low and high effort. The
production row uses Warden 0.46.2 on Pi with Grok 4.5 high for scanning and
GPT 5.6 Luna high for auxiliary calls; it records 37 of 86 known findings.
Add a source-linked production profile and require stable benchmark data to
contain exact matches for both model lanes. This keeps the matrix from
labeling a stale run or a primary-only match as current production. The
committed result files are sanitized summaries; raw JSONL remains withheld
pending sensitive-data review.
All six production shards completed without repair. The final auxiliary merge
dropped two exact scan-time corpus matches, so the published 37/86 is
intentionally the end-to-end system score rather than primary-model recall.
The Kimi rows preserve the completed and repaired shard provenance needed to
interpret their costs.
Validation passed with
pnpm lint,pnpm build,pnpm test,pnpm --filter warden-docs check, andgit diff --check.