Skip to content

Commit f981ed6

Browse files
committed
Fix: Validate sessionDefaults at config load time
Validate sessionDefaults immediately in readConfig instead of deferring until writeMcpConfig. This fails fast on unknown keys or invalid value types before expensive simulator setup, matching the documented behavior in the benchmark README.
1 parent ed3417c commit f981ed6

2 files changed

Lines changed: 4 additions & 1 deletion

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222

2323
### Fixed
2424

25+
- Fixed Claude UI benchmark suite loading so `sessionDefaults` unknown keys and invalid value types fail fast at config load time instead of after expensive simulator setup.
2526
- Fixed Claude UI benchmark preflight so transient malformed or still-loading UI snapshots no longer crash the harness or finish before app UI is observable.
2627
- Fixed Claude UI benchmark preflight so configured first-run dismissals require a concrete simulator ID and suite-provided simulator IDs are recorded in command logs.
2728
- Fixed Claude UI benchmark config handling so invalid `failurePatterns` regexes fail before a suite starts and partial `allowedVariance` overrides preserve defaults for omitted metrics.

src/benchmarks/claude-ui/config.ts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -185,7 +185,9 @@ export function readConfig(raw: unknown, source: string): BenchmarkConfig {
185185
),
186186
};
187187

188-
if (isRecord(raw.sessionDefaults)) config.sessionDefaults = raw.sessionDefaults;
188+
if (isRecord(raw.sessionDefaults)) {
189+
config.sessionDefaults = validateSessionDefaults(raw.sessionDefaults);
190+
}
189191
config.allowedVariance = readAllowedVariance(raw.allowedVariance, `${source}.allowedVariance`);
190192

191193
if (raw.baseline !== undefined) {

0 commit comments

Comments
 (0)