Skip to content

VU-1134: replace agent eval harness with tiered OpenCode runner#490

Merged
hbanerjee74 merged 15 commits into
mainfrom
feature/vu-1134-replace-agent-eval-harness-with-migration-utility-tiered
May 1, 2026
Merged

VU-1134: replace agent eval harness with tiered OpenCode runner#490
hbanerjee74 merged 15 commits into
mainfrom
feature/vu-1134-replace-agent-eval-harness-with-migration-utility-tiered

Conversation

@hbanerjee74

Copy link
Copy Markdown
Contributor

Fixes VU-1134

Summary

  • Replaced the app-local Claude-specific Promptfoo harness with a root tests/evals OpenCode/Promptfoo harness adapted from the migration-utility pattern.
  • Added suite-level eval tier/runtime configuration, deterministic contract tests, repo-local Promptfoo state handling, and worktree Promptfoo DB symlinking.
  • Removed the old app/agent-tests/promptfoo provider/config path and routed existing smoke commands through the new harness.
  • Added release/docs/CI follow-up work: deterministic release-stage and help-doc checkers, refreshed user guide pages, fixed release resource packaging, and made PR CI cheaper through targeted policy checks.
  • Updated README, AGENTS, CLAUDE, TEST_MANIFEST, repo-map, Dependabot, and policy workflows for the new eval and CI shape.

Validation

  • cd tests/evals && npm test
  • node --test scripts/*.test.mjs && node scripts/check-help-docs.mjs
  • bash app/scripts/lint-agent-docs.sh
  • Workflow YAML parse for .github/workflows/*.yml
  • node -e "JSON.parse(require('fs').readFileSync('repo-map.json','utf8')); console.log('repo-map json ok')"
  • cd docs && npm ci && npm run build
  • npx markdownlint-cli2 "README.md" "AGENTS.md" "CLAUDE.md" "TEST_MANIFEST.md" ".claude/rules/**/*.md" "docs/user-guide/**/*.md" "docs/superpowers/**/*.md"
  • cd app && npx tsc --noEmit
  • git diff --check origin/main...HEAD

PR phase gates

  • AC gate: pass. VU-1134 uses bullet ACs rather than checkboxes; all listed ACs are covered by committed work and validation above.
  • Design conformance: not_applicable. Linear issue has no linked local design document or attachment.
  • Promptfoo live evals: skipped by policy. Live/model-backed smoke commands are documented as manual and should be run by the user after selecting the target model.

Manual follow-up

After selecting the target model/runtime, run the live smoke command manually:

cd app && npm run test:agents:smoke

@linear-code

linear-code Bot commented May 1, 2026

Copy link
Copy Markdown

@hbanerjee74 hbanerjee74 force-pushed the feature/vu-1134-replace-agent-eval-harness-with-migration-utility-tiered branch from c643c0c to cf6c39c Compare May 1, 2026 07:31
@hbanerjee74 hbanerjee74 merged commit 7f64d6a into main May 1, 2026
14 checks passed
@hbanerjee74 hbanerjee74 deleted the feature/vu-1134-replace-agent-eval-harness-with-migration-utility-tiered branch May 1, 2026 07:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant