feat(hdx-eval): add scenario hooks for custom prompts, tools, grading, and inspection

[brandon-pereira]

Summary

Adds optional hooks to the Scenario type so new eval scenario kinds can customize harness and grading behavior without modifying framework files. All existing scenarios are unaffected — hooks are optional with defaults preserved.

Hooks

Hook	Purpose
buildSystemPrompt(ctx)	Replace the default SRE-investigation system prompt
allowedToolPatterns	Selectively unblock denied MCP tools (e.g. dashboard tools)
judgeSystemPreamble	Replace the default LLM judge preamble
postRunInspection(ctx)	Inspect artifacts post-run, collect evidence for the judge, clean up

Why

Adding the dashboard-build eval (separate PR) required changes to 10+ framework files because ScenarioKind was threaded through the entire call stack. With hooks, adding a new scenario kind requires only the scenario files + one import line in index.ts.

Other changes

• runCell takes a Scenario object instead of a scenario name string (needed for hook dispatch)
• GradeRecord has generic inspectionSummary and inspectionEvidence fields
• Re-grade safety: cached inspection summary + evidence are reused on --rerun-judge instead of re-firing the hook (artifacts may have been cleaned up)
• SystemPromptContext includes variant so custom hooks can adapt to hypothesis-mode runs
• buildInspectionConfig() helper extracted (used by both run and grade commands)
• CLI: --email/--password options on run and grade commands (extracted to DEFAULT_EVAL_EMAIL/DEFAULT_EVAL_PASSWORD constants)
• .gitignore: ignore eval.config.*.json backup configs

Test plan

• 149 unit tests pass (16 suites)
• TypeScript compiles clean
• All existing scenarios produce identical results (hooks are optional, defaults preserved)
• New test: buildSystemPrompt hook dispatch verified with a fake scenario

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Jul 1, 2026 2:52pm
hyperdx-storybook	Ready	Preview, Comment	Jul 1, 2026 2:52pm

[github-actions]

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

• Diff size: 391 production lines changed (Tier 2 max: < 250)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats

• Production files changed: 11
• Production lines changed: 391 (+ 33 in test files, excluded from tier calculation)
• Branch: brandon/eval-scenario-hooks
• Author: brandon-pereira

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[greptile-apps]

Greptile Summary

This PR introduces an optional hook system on the Scenario type (buildSystemPrompt, allowedToolPatterns, judgeSystemPreamble, postRunInspection) so new eval scenario kinds can customize harness and grading behavior without touching framework files. All existing scenarios are unaffected — hooks default to the existing SRE-investigation behavior.

• Hook dispatch: systemPrompt.ts, settingsFile.ts, judge.ts, and grade.ts each check for the relevant hook and fall back to the original logic when absent; variant is correctly forwarded to SystemPromptContext and anchorTimeIso is now included in the grade command's inspection config.
• Re-grade safety: gradeOne reuses the cached inspectionSummary/inspectionEvidence from an existing grade record before calling the hook, so --rerun-judge works even after artifacts have been cleaned up.
• CLI ergonomics: DEFAULT_EVAL_EMAIL/DEFAULT_EVAL_PASSWORD are extracted as constants; --email/--password options are added to both the run and grade commands; buildInspectionConfig is extracted to avoid duplication.

Confidence Score: 4/5

Safe to merge for existing scenarios; new hook infrastructure works correctly for the baseline case.

The hook dispatch, re-grade cache, and CLI wiring are all correct. The one remaining concern is that formatInspectionLogBit in grade.ts is dashboard-specific and will silently produce empty log output for any future non-dashboard inspection hook, making the console output misleading as the hook system is adopted by more scenario types.

packages/hdx-eval/src/grading/grade.ts — formatInspectionLogBit and the silent skip when hyperdxApi is absent.

Important Files Changed

Filename	Overview
packages/hdx-eval/src/scenarios/types.ts	Adds four optional hook fields to the Scenario type plus supporting context/result types; design is well-documented and backward-compatible.
packages/hdx-eval/src/grading/grade.ts	Adds post-run inspection logic with cache-reuse guard for re-grades; formatInspectionLogBit is coupled to dashboard-specific summary fields.
packages/hdx-eval/src/harness/settingsFile.ts	Adds allowedToolPatterns filtering to deniedToolsFor and buildSettings; correctly applies only to per-MCP denied tools.
packages/hdx-eval/src/cli.ts	Extracts credential constants, adds --email/--password to run and grade commands, and adds buildInspectionConfig helper; anchorTimeIso is no longer dropped in the grade command.
packages/hdx-eval/src/harness/systemPrompt.ts	Dispatches to scenario.buildSystemPrompt hook when present; variant is correctly forwarded to the custom hook via SystemPromptContext.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant CLI as cli.ts
    participant RR as runRun.ts
    participant SP as systemPrompt.ts
    participant CS as claudeSpawn.ts
    participant SF as settingsFile.ts
    participant GB as gradeBatch
    participant GO as gradeOne
    participant JT as judge.ts

    CLI->>RR: runCell(scenario, ...)
    RR->>SP: buildSystemPrompt(scenario, anchorTimeIso, variant)
    SP-->>SP: scenario.buildSystemPrompt?(ctx) or default
    SP-->>RR: systemPromptAppend
    RR->>CS: "runClaude({allowedToolPatterns, ...})"
    CS->>SF: deniedToolsFor(variant, def, allowedToolPatterns)
    SF-->>CS: filtered deny list
    CS-->>RR: SpawnResult
    RR-->>CLI: RunRecord

    CLI->>GB: "gradeBatch(batchDir, {inspectionConfig})"
    GB->>GO: gradeOne(record, existing, opts)
    alt existing.inspectionSummary present
        GO-->>GO: reuse cached inspectionResult
    else scenario.postRunInspection and inspectionConfig
        GO->>GO: scenario.postRunInspection(ctx)
    end
    GO->>JT: "judgeTrajectory({judgeSystemPreamble, inspectionEvidence})"
    JT-->>GO: JudgeResult
    GO-->>GB: GradeRecord
    GB-->>CLI: GradeBatchSummary

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant CLI as cli.ts
    participant RR as runRun.ts
    participant SP as systemPrompt.ts
    participant CS as claudeSpawn.ts
    participant SF as settingsFile.ts
    participant GB as gradeBatch
    participant GO as gradeOne
    participant JT as judge.ts

    CLI->>RR: runCell(scenario, ...)
    RR->>SP: buildSystemPrompt(scenario, anchorTimeIso, variant)
    SP-->>SP: scenario.buildSystemPrompt?(ctx) or default
    SP-->>RR: systemPromptAppend
    RR->>CS: "runClaude({allowedToolPatterns, ...})"
    CS->>SF: deniedToolsFor(variant, def, allowedToolPatterns)
    SF-->>CS: filtered deny list
    CS-->>RR: SpawnResult
    RR-->>CLI: RunRecord

    CLI->>GB: "gradeBatch(batchDir, {inspectionConfig})"
    GB->>GO: gradeOne(record, existing, opts)
    alt existing.inspectionSummary present
        GO-->>GO: reuse cached inspectionResult
    else scenario.postRunInspection and inspectionConfig
        GO->>GO: scenario.postRunInspection(ctx)
    end
    GO->>JT: "judgeTrajectory({judgeSystemPreamble, inspectionEvidence})"
    JT-->>GO: JudgeResult
    GO-->>GB: GradeRecord
    GB-->>CLI: GradeBatchSummary

Loading

_{Reviews (6): Last reviewed commit: "Merge branch 'main' into brandon/eval-sc..." | Re-trigger Greptile}

[changeset-bot]

🦋 Changeset detected

Latest commit: b190e46

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@hyperdx/hdx-eval	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[github-actions]

E2E Test Results

✅ All tests passed • 223 passed • 3 skipped • 1509s

Status	Count
✅ Passed	223
❌ Failed	0
⚠️ Flaky	3
⏭️ Skipped	3

Tests ran across 4 shards in parallel.

View full report →