Skip to content

v0.1.0 — initial release

Choose a tag to compare

View all tags
@hoainho hoainho released this 28 May 15:31
· 62 commits to main since this release
v0.1.0
9eb4b05

Initial release

Behavior-regression eval harness for opencode skills.

Scope (v0.1.0)

  • Structured-output skills only. Prose-output skills (pr-code-reviewer, od-workflow, blog-workflow, idea-workflow) deferred to v0.3 (LLM judge).
  • Deterministic mode (T=0, k=1). pass@k deferred to v0.2.
  • 3 triggers: manual via eval-harness run, git pre-push, sync-skill-to-manager pre-publish (opt-in per skill).
  • opencode 1.15.10 verified. Earlier/later versions: file an issue.

Architecture

  • Bash + jq + python3 (yq-shim for YAML). No daemon, no Unix socket, no Node CLI binary.
  • 4-class attribution: SKILL_CHANGED / FIXTURE_STALE / MODEL_CHANGED / UNKNOWN_DRIFT.
  • 6-field FAIL schema: failed_check_id, expected, actual, diff_hint, transcript_span (nullable), env_delta.
  • One-command rerun in every FAIL output.
  • Warn-only by default for 7 days. eval-harness promote enables blocking.

Safety rails

  • Two-stage accept: default keeps env_manifest; --bless-env requires confirmation (avoids silent model drift).
  • EVAL_BYPASS=1 escape hatch with audit log to history.ndjson.
  • Ephemeral sandbox per case (fresh HOME, OPENCODE_CONFIG_DIR, NANO_BRAIN_ROOT, cwd).

Design provenance

Designed via the deep-design multi-agent pipeline:

  • Phase 1: Metis (scope/risk) + Oracle (architecture) in parallel
  • Phase 1.5: cross-critique (Metis ↔ Oracle)
  • Phase 2: confidence-scored synthesis → 18 Settled Decisions
  • 15 of 18 fully implemented; 3 partial implementations documented in README.

Demo

npm test

Runs scripts/eval/tests/regression_inject.sh end-to-end. All 5 acceptance assertions green:

  • verdict=REGRESSION
  • attribution=SKILL_CHANGED
  • regression list contains atom-tags-decision-architecture
  • failed_check_id populated
  • warn-only exit 0 (not yet promoted)

Known limitations

  • 3-sample stability check implemented but not wired into critical path (deferred to v0.2)
  • pricing.json dollar conversion + staleness gate deferred to v0.2
  • Per-(case,trigger) lockfile deferred to v0.2 (only needed when Stop hook lands)

Roadmap

  • v0.2.0: opencode Stop hook + per-repo opt-in registry + lockfile coordination
  • v0.3.0: LLM judge (claude-haiku, cross-model debias, 3-sample majority) + pr-code-reviewer demo
  • v0.4.0: heuristic auto-fix proposer (constrained to literal/regex checks)

Roadmap-adjacent: skill-reviewer

Separate skill (not in this repo) consuming standards/skill-quality-v1.md for design review (Job 3). Eval-harness measures BEHAVIOR REGRESSION; skill-reviewer measures DESIGN QUALITY.

See README.md for full installation + usage.

Assets 2
Loading