unbound-force
Replies: 10 comments 9 replies
Addendum: Configuration Management — Putting OpenCode Config Under Source ControlThe ProblemThe token-efficiency-relevant configuration currently lives in places that are not under source control for the org:
The model routing, plugin selection (hashline, DCP), agent model assignments, and compaction settings — the exact things we want to evaluate for token efficiency — all live in the global user config. They are invisible to CI, invisible to the eval workflow, and not reproducible on another machine or by another developer. Current State AuditAn audit of all 7 repos under
How OpenCode Config Resolution WorksOpenCode uses a deep merge chain with this precedence (later = higher priority):
Project-level configs merge with global config — they do not replace it. This means a project Three OptionsOption A: Move token-relevant config into each repo's
|
Phase 1 Complete — Implementation StatusPhase 1 (Foundation) from the original proposal is implemented in Delivered
Not Yet Validated
Constraint: eval-infra Lives in Personal OrgThe plan assumed eval-infra would land in Secrets Audit
Only Refined Testing Strategy: Smoke vs. IntegrationDuring implementation, a useful split emerged that wasn't in the original proposal:
Smoke tests are fast and gate PRs. Integration tests are slower and monitor drift over time (catching regressions from upstream model updates, SDK changes, etc.). The hybrid approach:
Adjusted Next StepsGiven the WIF constraint, the original rollout phases need reordering: Step A: Local Smoke Test Experiment (no CI needed) ← Starting hereWrite a single smoke fixture for a simple command and run it locally from the eval-infra driver pointing at cd /path/to/eval-infra/driver
bun run-eval.ts \
--fixtures /path/to/unbound-force/.eval/fixtures \
--model google-vertex-anthropic/claude-opus-4-6@default \
--repo unbound-force/unbound-force \
--output /tmp/eval-outThis validates the full driver pipeline end-to-end (SDK session → token capture → SQLite → summary) before touching any CI config. Requires only local opencode + GCP credentials (which developers already have via Step B: CI Smoke Test in unbound-force/unbound-forceSince
Step C: Phase 0 — Config Source ControlMove token-relevant config (model, plugins, agent routing, compaction) into each repo's Step D: Auth Fallback — Support OPENCODE_API_KEYUpdate the reusable workflow to support both GCP WIF and direct API key auth. This unblocks Step E: Transfer eval-infra to unbound-forceOnce the pipeline is validated end-to-end via Steps A-B:
Step F: Phase 2 — First Consumer FixturesWrite real fixtures for high-value commands in
Step G: Phase 4 — Expansion to Other ReposOnboard |
Step A Complete — Smoke Test Validated End-to-EndThe first smoke test ( CI Run Results
What the fixture testsThe smoke test runs Reporting ImprovementsTwo reporting features were added to the reusable workflow:
Bugs Fixed During Validation
Key Design Decision: Fixture as Project RootEach fixture directory is the opencode project root for that eval run. The driver starts opencode inside the fixture directory, so commands find their expected file structure ( What's on each branch
Next StepsPer the adjusted plan, the next actions are:
|
|
@jflowers Adding here a few design edge cases worth considering as this moves into Phase 2+, let me know what you think:
|
|
This are all great observations! I agree with all of it. |
Design Discussion: Complex Fixture Support (Setup/Teardown)The current fixture format works well for self-contained commands like But commands like The Problem
Similar requirements for Proposed Solution: Setup Executors + Dedicated Test RepoDedicated test subject repo: Extended fixture format with a # task.yaml
task_id: review-pr-smoke
task_version: "1"
command: uf.review-pr
timeout_seconds: 300
setup:
type: github-pr
repo: unbound-force/eval-test-subject
base_branch: main
eval_branch: eval/review-pr-smoke # static branch name (the known diff)
patch: patch.diff # deterministic diff in fixture dir
pr_title_prefix: "[EVAL-${RUN_ID}]" # dynamic PR name for concurrency safety
pr_body: "Automated eval fixture — do not merge."
verify:
- type: pr-has-review
expect: true
- type: pr-comment-contains
pattern: "## Summary"Key design decisions:
Concurrency Model
Incremental AdoptionThis doesn't need to be built all at once:
Open Questions
|
Decision: Should Eval Fixtures Use a Test Framework?As we build out more complex fixtures (setup/teardown for The Problem
Existing test frameworks already provide all of this with mature ecosystems, IDE integration, CI tooling, etc. What's Actually Unique to EvalOnly these pieces are domain-specific and would remain as a library regardless of framework choice:
Everything else is commodity test infrastructure. Framework CandidatesIf we adopt a framework, the two realistic options are bun:test (already in use for unit tests) and Vitest (dominant JS/TS test framework). Comparison
What Eval Fixtures Would Look LikeWith bun:test (minimal change from current): import { describe, it, expect, beforeAll, afterAll } from "bun:test"
import { EvalFixture } from "./eval-fixture"
describe("uf.constitution-check", () => {
const fixture = new EvalFixture("./fixtures/uf-constitution-check")
beforeAll(async () => await fixture.setup())
afterAll(async () => await fixture.teardown())
it("completes successfully", async () => {
const result = await fixture.run()
expect(result.exitStatus).toBe("success")
})
it("captures token metrics", () => {
expect(fixture.getTokens().cost_usd).toBeGreaterThan(0)
})
it("stays within baseline cost", () => {
expect(fixture.compareBaseline().status).not.toBe("regression")
})
})With Vitest (same test code, but with additional capabilities): // vitest.config.ts — separate project for eval
export default defineConfig({
test: {
projects: [
{ name: 'unit', include: ['__tests__/**'] },
{ name: 'eval', include: ['.eval/**/*.test.ts'], globalSetup: './eval-setup.ts' },
],
reporters: ['default', './eval-reporter.ts'], // custom eval summary reporter
}
})Arguments For Adopting a Framework
Arguments For Keeping
|
|
There are a lot of positives from setting up eval-infra as suggested/done above.
A few items to consider
Open questions from the thread:
|
|
Still running through things, but wanted to drop in some prior art that may be useful:
|
|
In an ideal world, we would be able to cleanly decouple measurements for each part of the system and attribute measurements to different models, versions of included skills, etc... This was the idea behind the baseline + model + agent (opencode/claude code) multiplexing in To answer your question, my hope would be that the larger, flexible, measurement harness would fully encompass the needs of any agentic framework in a way that all you need is minor scaffolding and prompt/multi-turn drivers to proceed. That's (poorly) working in Nicely, all of these systems are just state machines with a combination of deterministic, and quasi-deterministic, steps. The deterministic portions can be tested internally but the quasi-deterministic portions require a judge model with the ability to target a reasonable range of allowable stochastic deviation. Below is my original Claude-driven deep dive between lola-eval vs agent-eval-harnesslola-eval vs. agent-eval-harness: A Deep Comparison
Table of contents
1. TL;DRBoth tools score runs of agent CLIs with an LLM-as-judge and gate on thresholds. They are nonetheless built for different jobs and sit at opposite ends of the "evaluation as a developer loop" vs. "evaluation as a CI gate" spectrum.
If you are iterating on a Claude skill and want help fixing it, reach for agent-eval-harness. If you are gating packs/agents in CI and need reproducible, auditable score history, reach for lola-eval, with the maturity caveat in §19. 2. What each tool isagent-eval-harnessA generic evaluation framework for agents and skills, delivered as a Claude Code plugin. The user-facing surface is seven skills that orchestrate thin Python helper scripts: The spine is The core abstraction is a lola-evalAn embeddable test runner that verifies lola packs still produce useful results when run through It is designed to be embedded in a target repo like any other test suite, with results (
3. At-a-glance matrix
4. Shared DNA (where they genuinely overlap)Despite very different shapes, the two are conceptual cousins:
Everything below is where they diverge. 5. Distribution & packagingagent-eval-harness: a plugin, deps on demand
The net effect is a light footprint that grows lazily; you pay for MLflow or Anthropic only when your eval uses them. lola-eval: a fat, hermetic RPM
Trade-off: lola-eval buys total reproducibility (identical Python/Node/ promptfoo everywhere) at the cost of a 2 GB footprint and an RPM-only world. agent-eval-harness buys a light footprint and easy install at the cost of depending on the host's Python/Claude environment. 6. Execution / runner architectureThis is one of the sharpest architectural differences. agent-eval-harness: own runner, strategy/registry pattern
lola-eval: promptfoo wrapper, providers + assertion
Summary: AEH owns the subprocess and reimplements orchestration; lola-eval delegates orchestration to promptfoo and only re-takes control inside the assertion callback. AEH's runner is more capable on the 7. Client & agent supportWhich agent CLIs / model providers can be the subject under test, which can be the judge, and how hard it is to add a new one. This is where many adoption decisions actually land. Subjects under test
Key facts behind the table:
Judge clients/providers
Interactive / multi-turn
Cost of adding a new client
Other clients (cursor / windsurf / gemini-cli / codex / copilot / …)Neither tool supports any of these as a client. Specifically:
Bottom line: AEH's implemented subjects are Claude, OpenAI, and "any CLI at low fidelity"; lola-eval's are Claude and opencode at full fidelity, plus real multi-turn. AEH is broader on providers (OpenAI) and easier to extend cheaply; lola-eval is deeper on opencode and dialogue but locked to two clients. 8. Configuration philosophy
agent-eval-harness's schema-driven design is a first-class constitutional principle ("Dataset and output structures are described in natural language… no extraction spec and no hardcoded field names", lola-eval is the opposite: every field is typed and validated at load. The 9. Datasets & test-case authoringA capability that is entirely one-sided. agent-eval-harness: generates datasets
Upstream, lola-eval: bring your own fixturesThere is no dataset generation. Contrast: AEH can bootstrap you from zero and grow coverage from production usage (LLM-authored, less deterministic); lola-eval expects curated fixtures (more manual, but fully deterministic and reviewable). 10. Judges & scoringagent-eval-harness: four judge types, programmableScoring lives in
Plus orthogonal features: The Aggregation: all-bool judges → lola-eval: one judge, one weighted composite, consensus across judgeslola-eval has a single judge type: an LLM trajectory judge scored against a weighted rubric.
Contrast
AEH is the more programmable judge layer; lola-eval is the more statistically disciplined one. 11. Thresholds, baselines & regressionagent-eval-harness
lola-evalThree threshold modes (
Contrast: AEH's regression check is a coarse "−0.5 from baseline" heuristic on per-judge aggregates; lola-eval's is a precise per-cell 12. Drift, trend & comparison analyticsThis is where the tools diverge most in philosophy. agent-eval-harness: delegate to MLflowAEH has no native history store. Trend analysis happens in MLflow: runs are logged as experiments, datasets are synced, and traces accumulate. There is no lola-eval: a first-class, fingerprinted analytics suitelola-eval persists every run to a SQLite
Contrast: lola-eval treats score history as the product; agent-eval-harness treats the current run as the product and outsources history to MLflow. 13. Cost handlingagent-eval-harness: caps during execution
So AEH's cost story is "cap the spend and record what was spent." lola-eval: predict before you spend
Calibration data is refreshed from real runs ( Contrast: AEH manages cost reactively (cap + record); lola-eval manages it proactively (forecast the whole matrix before committing money). 14. Telemetry, tracing & deploymentagent-eval-harness: MLflow + a Kubernetes provider
lola-eval: self-contained artifacts, no external servicelola-eval emits everything locally: Contrast: AEH is built to integrate with an MLOps platform; lola-eval is built to stay inside the repo and CI with zero external infrastructure. 15. The evaluation matrix & profilesagent-eval-harnessThere is no combinatorial matrix concept. You evaluate one skill across its cases, with one runner/model (CLI flags can override the model). Variation across agent configurations is not a built-in axis. lola-evalThe matrix is target × model × pack × profile × case, enumerated as nested loops (
Contrast: lola-eval's profile axis is purpose-built for "does adding framework/skill X to the agent help or hurt?" across many configurations; something agent-eval-harness simply doesn't model. 16. Closed-loop improvementThis is agent-eval-harness's signature capability, and lola-eval has no equivalent.
Guardrails are explicit: never modify judges, lola-eval is the deliberate opposite: a measurement instrument that scores, fingerprints, and gates, but never touches the pack/skill/agent under test. 17. Security modelagent-eval-harnessSecurity is handled per-runner and via tool interception rather than a single threat-model document:
lola-evalA dedicated
Contrast: lola-eval has a more formalized threat model (published doc, explicit guards, credential-handling audit trail), reflecting its run-on-real-credentials posture; AEH's security is distributed across runner implementations and the tool-interception layer, with a trusted-config assumption for inline judges. 18. CI integration
lola-eval is CI-native: a single 19. Project health, maturity & governance
Methodology, restated: agent-eval-harness is spec-kit / spec-driven ( 20. Dependency licensingBoth projects' own agent-eval-harness
Notable: MLflow is Apache-2.0 (not a copyleft/"server-side" license), and eval-hub-sdk is public and Apache-2.0 (not a private/closed dependency). Clean. lola-eval
The one adopter action item: because lola-eval redistributes promptfoo's entire bundled 21. Documentation-vs-code discrepanciesThe deep-verification pass found places where each project's prose overstates or misdescribes its code. These are worth knowing before relying on a feature. agent-eval-harness
lola-eval
22. Decision guideChoose agent-eval-harness when:
Choose lola-eval when:
They can also be complementary: use agent-eval-harness's optimize loop to develop and improve a skill/pack, then use lola-eval to gate and trend-track it in CI across model versions and configurations, accepting that you may be maintaining lola-eval yourself. 23. Appendix: evidence mapKey claims and where to verify them in the cloned trees. agent-eval-harness (
|
| Topic | File:line |
|---|---|
EvalRunner ABC + RunResult |
agent_eval/agent/base.py:14-65 |
RUNNERS registry (claude-code, cli, responses-api) |
agent_eval/agent/__init__.py:8-12 |
runner.type free-form + execute-time check |
agent_eval/config.py:122,280; skills/eval-run/scripts/execute.py:99-104 |
claude-code: claude --print, budget, perms |
agent_eval/agent/claude_code.py:108-140 |
| responses-api → OpenAI Responses API | agent_eval/agent/responses_api.py:50,85-96,306-317 |
| Opaque CLI runner contract + limitations | docs/opaque-cli-runner-contract.md:9-24,141-151 |
| case/batch modes, parallelism (threads) | skills/eval-run/scripts/execute.py:142,350-381 |
| AskUserQuestion 3-tier answering + hook model | skills/eval-run/scripts/tools.py:103-212 |
| Dataset generation strategies | skills/eval-dataset/SKILL.md (bootstrap/expand/from-traces) |
| Four judge types dispatch | skills/eval-run/scripts/score.py:361-419 |
| Judge providers (Anthropic/Vertex/MLflow) | skills/eval-run/scripts/score.py:687-742,867-877 |
Inline check: execution (full builtins) |
skills/eval-run/scripts/score.py:645-658 |
| Pairwise (position-swapped) | skills/eval-run/scripts/score.py:783-856 |
| Thresholds / regression | skills/eval-run/scripts/score.py:964-996 |
| Hierarchical MLflow trace builder | agent_eval/mlflow/trace_builder.py:51-821 |
| EvalHub adapter | agent_eval/evalhub/adapter.py:161-409 |
| Dep bootstrap (SessionStart) | scripts/ensure_deps.py:35-87; agent_eval/_bootstrap.py:20-38 |
| eval-optimize loop + guardrails | skills/eval-optimize/SKILL.md |
| Schema-driven constitution | .specify/memory/constitution.md:5-6 |
lola-eval (/tmp/lola-eval)
| Topic | File:line |
|---|---|
| promptfoo config gen + subprocess | src/lola_eval/runner.py:124-129,290-315 |
| Agent CLIs as JS providers | src/lola_eval/_data/providers/claude_code_provider.js:147-193; opencode_provider.js:141-160 |
cli is a hard enum (targets/judge/sim-user) |
src/lola_eval/config.py:21,37,45; tests/python/test_config.py:159-169 |
| Provider/interactive maps + add-client surface | src/lola_eval/runner.py:437-464; _data/providers/lib/interactive_helper.js:37-81 |
| Judge as promptfoo py-assert | src/lola_eval/_data/judges/trajectory_judge.py:305 |
| Judge via CLI subprocess + stdin | src/lola_eval/judge_client.py:166-251 |
| Interactive multi-turn dialogue | src/lola_eval/_data/interactive/orchestrator.py:124-231 |
| Tool registry (config_dir/env; permission_flag unused) | src/lola_eval/_data/tools.json; provider hardcodes flag |
| Matrix enumeration | src/lola_eval/runner.py:498-519 |
| Mode 1/2 + pack sentinels | src/lola_eval/runner.py:89-91; config.py:240-254 |
| Profile schema + inheritance | src/lola_eval/profile.py:37-108 |
| Fingerprint v2 composition | src/lola_eval/fingerprint.py:27,54-67 |
runs.db schema + provenance |
src/lola_eval/store.py:9-83 |
| Composite math (clamped) + consensus | src/lola_eval/judge.py:22-72 |
| Disagreement (pstdev) | src/lola_eval/judge.py:69,73; trajectory_judge.py:491-516 |
| Transcript head+tail truncation | src/lola_eval/judge_client.py:85-135 |
| Threshold modes + exit codes | src/lola_eval/threshold.py:3,170-222 |
| lift / drift formulas | src/lola_eval/math.py:13-24; report.py:33-113 |
| compare-ref git-worktree safety | src/lola_eval/compare_ref.py:49-116 |
| profile-compare conflict logic | src/lola_eval/profile_compare.py:82-110 |
| Three-tier cost + kNN | src/lola_eval/cli/test_cmd.py:95-177; calibration.py:283-457 |
| reset.sh XDG guard | src/lola_eval/_data/.../reset.sh:64-72 |
| Subscription-auth credential copy | src/lola_eval/_data/providers/lib/profile_setup.js:126-143 |
| RPM pinned versions | packaging/rpm/lola-eval.spec:77-114 |
| doctor bundle verification | src/lola_eval/cli/doctor_cmd.py:113-368 |
Generated 2026-06-01. Code verified against agent-eval-harness 5814e61 and lola-eval 62a26d4; project-health/licensing via GitHub/PyPI/npm APIs same day. Static analysis only; neither tool was executed. Where README and code disagreed, code was treated as authoritative and the discrepancy recorded in §21.
Absolutely! When the framework doing the judging changes, that needs to essentially reset the baseline until it's determined that the data is within a standard deviation. |
|
I would expect provider/model changes to be an occasional migration event with some pre-testing. So much of it depends on the cost-benefit ratio that it's hard to predict honestly. I think that multi-provider is something that would be hard to add later and would likely cause a rewrite if not taken into consideration early (well, it did when I was working on |
|
Following up on the prior art I dropped earlier along with a proposal for moving forward. TL;DR: After a deep dive with our good 🤖 friend Claude, I think that consolidation around agent-eval-harness (AEH) makes the most sense for a "build better together" resolution. I've put this together with a lot of
Proposed targets
My project, lola-eval, gets pretty much completely replaced by AEH 🎉 outside of a small Lola-specific wrapper. Fixtures, the caller workflow, and opencode config profiles belong in unbound-force. lola-pack install, the pack matrix, and skill-conflict detection stay in lola-eval, since they are specific to my packs rather than to evaluation. Two AEH issues would unblock most of this. One covers targets 2 through 4, which are three independent commits against code that already exists. One covers target 6. What AEH already has, so we stop rebuilding it
In flight and relevant here:
Several of these overlap directly with eval-infra's Phase 1 through 4. #90 and #67 are drafts, so they are waiting on their author rather than on reviewers. What is still missing
Question handling: policy versus transportThe policy is which answer a question gets. Pinned answers for questions the fixture knows about, something else for the ones it does not, and a class of question that should stop the run rather than be answered. Plus a record of every question asked and how it resolved. None of that is runner-specific. The transport is how you see a question mid-turn and write a reply back into a live session. That is entirely runner-specific. Claude Code uses a AEH already has a good policy, including three-tier resolution with per-case overrides. It lives inside lola-eval has neither. Its provider splices So the proposal is to lift the policy out and define a small contract runners implement. Then there are three transports in increasing order of capability, each useful on its own:
The opencode transport does not need the TS SDK
The relevant endpoints are already documented: That matters for AEH specifically. A Python harness can spawn Needs confirmation: which of those channels carries a user-facing question as opposed to a permission request. Note also that A useful side effect: once questions are recorded, Run identity needs no new storageAEH writes every run to The proposed resolution is to add Plenty of other things invalidate a comparison too, which argues for recording more while keying on less. A harness upgrade, a new agent CLI version, a changed timeout. If all of that went into the identity hash, every container rebuild would fork a new series and you would end up with many series of length one. So the environment gets recorded as a manifest alongside, with its own Three properties that matter:
If walking the tree ever gets slow, a derived index is the obvious cache, but it should stay regenerable rather than authoritative. Why the ask is deliberately smallAEH's throughput is concentrated. One maintainer accounts for about 70 percent of its commits and authored all five of the PRs listed above. So the two issues I would file are both small, and both are shaped to be reviewable in one sitting. Targets 2 through 4 are a refactor with no behaviour change on the path AEH already ships, plus two opt-in additions. Target 6 is a few fields and a hashing module. Neither depends on anyone's unreleased work. It also means converging on AEH buys less bus-factor relief than it looks like from outside, which is worth being honest about given that reducing my own bus factor is part of why I want this. Order of work, and what would stop itOrdered by dependency. Nothing here binds anyone who has not agreed.
Main risks: no maintainer picks up review, so the work competes with one person's own backlog; the pricing snapshot turns out not to be redistributable; #172 never lands. The Future: A shared cost corpuslola-eval forecasts cost before a run by comparing against a record of what similar runs actually cost. I am not proposing to carry forward that record because it was collected against my own packs through my own harness, so it describes my workloads rather than anything general, and seeding a shared thing with one project's data starts it off skewed. The machinery is worth contributing. The corpus is worth building in the open. If the harness records what a given shape of task cost, and people contribute those records back, you get something none of us can produce alone. A reference for what evaluation actually costs, and for which patterns cost more than they should. That is a learning resource as much as a budgeting one. New users could see what a reasonable run looks like before spending anything, and the rest of us could see which of our habits are expensive. What lola-eval would contribute, and its liabilitiesBehind the small asks sit roughly 3,018 lines across fifteen modules. They cover identity and history, drift and comparison analysis, cost forecasting and calibration, judging, and the dialogue orchestrator. Code only; lola-eval's own calibration data is deliberately not offered. The liabilities belong in the same breath. lola-eval is pre-1.0 with no tagged releases, no CI on push or PR, one maintainer, and no rate-limit or backoff handling anywhere in its source. The end state for lola-eval is a thin lola-pack shim: pack install, the pack matrix, and skill-conflict detection. Everything else gets deleted once its replacement has merged and released. Method, and what I have not verifiedAEH claims come from the GitHub API and from raw file contents on Not verified at runtime: nothing here was executed. "Verified" means confirmed in source, in API output, or in the published docs via LLM and human spot checking. |
Uh oh!
There was an error while loading. Please reload this page.
Summary
Create a new
unbound-force/eval-infrarepository that provides:This enables ongoing, automated evaluation of both token cost and output quality for every command we ship.
Motivation
We have 47+ slash commands, 18 agents, and 15 skills across the org. Today there is no systematic way to answer:
Manual spot-checking doesn't scale. We need automated, repeatable evaluation that runs in CI and produces comparable data across time.
Prior Art Reviewed
complytime/org-infra
The complytime org uses a central
org-infrarepo withreusable_*.ymlworkflows consumed by other repos viaworkflow_call. Thereusable_crapload_analysis.ymlworkflow is the closest analog — it runs analysis against a committed baseline, produces structured outputs (pass/fail, counts, regressions), uploads artifacts, and leaves PR comments. The pattern of typed inputs/outputs, artifact upload, and a separate comment-posting job is directly applicable.complytime-labs/lola-eval
The most sophisticated open-source eval harness for opencode. Key concepts worth adopting:
runs.dbSQLite storestep_finishevent parsingtokens.input,tokens.output,tokens.cache.read/write,costfrom opencode JSON transcriptCritical gap in lola-eval: It drives opencode via
opencode run --format json --autoas a subprocess. The--autoflag approves tool permissions but cannot answerAskUserQuestionprompts. Unbound-force commands heavily use interactive questions throughout multi-phase pipelines. lola-eval's invocation model would stall and timeout on our commands.unbound-force/scripted (local, batch-fix)
A working TypeScript headless driver for opencode built on
@opencode-ai/sdk/v2. Key capabilities:session.status { type: "idle" }answers.ts) — 40+ regex patterns mapping questions toreply | bail | doneactionsclient.session.command()— proper slash command API invocation (not prompt text injection)This is the only approach that can drive unbound-force's interactive, multi-phase commands. The driver code is the foundation.
unbound-force/gaze CI workflow
The existing
gaze/.github/workflows/test.ymlproves the infrastructure stack:ubuntu-latestrunners (no self-hosted needed)npm install -g opencode-ai@1.2.26for opencode in CIHAS_GCP_SECRETS) so forks/repos without credentials skip LLM stepsThis means the entire auth + runner + opencode infrastructure is already in place.
Architecture
Repository Structure
Consuming Repo Structure
Repos that publish opencode commands/agents/skills add:
Reusable Workflow Contract
Reusable Workflow Steps
The
reusable_opencode_eval.ymlworkflow does:eval-infra(sparse checkout,driver/only) — same pattern as complytime's.workflow-scriptsbun driver/run-eval.ts --fixtures <path> --baseline <path> --model <model>runs.dbdelta, eval summary markdown, transcriptsstatus(pass/fail),composite-score,total-cost,total-tokensA separate job in the caller handles PR comments (same as complytime's
post-commentpattern).What Gets Measured
Token Efficiency (quantitative)
Per session, captured from opencode
step_finishevents:tokens_inputstep_finish.tokens.inputtokens_outputstep_finish.tokens.outputtokens_cache_readstep_finish.tokens.cache.readtokens_cache_writestep_finish.tokens.cache.writecost_usdstep_finish.costturnsstep_starteventstool_calls_counttool_useeventsduration_sOutput Quality (rubric-based LLM judge)
Each fixture includes a
rubric.mdwith weighted criteria. Example for/opsx-propose:The judge receives the agent's transcript + workdir diff + rubric, scores each criterion 0.0-1.0, and computes a weighted composite. Both the composite score and the token data are stored in
runs.dbso you can see cost-vs-quality tradeoffs.Drift Detection
Each row in
runs.dbgets a fingerprint:sha256(task_id, task_version, rubric_version, profile_id, ...). Rows with the same fingerprint are directly comparable. When a baseline exists, the workflow reports:runs.dbSchemaAdapted from lola-eval's schema, with additions for the SDK driver:
Execution Modes
Mode 1: Autonomous (
--auto)For commands that don't ask questions. Uses
opencode run --format json --autoas a subprocess (simpler, faster). Token data parsed from the JSON event stream.Mode 2: Interactive (SDK driver)
For commands that ask
AskUserQuestionprompts. Uses the@opencode-ai/sdk/v2SSE driver with pattern-based auto-response fromanswers.ts+ per-fixtureanswers.yamloverrides. This is the mode required for most unbound-force commands.The fixture's
task.yamldeclares which mode:Config Variant Testing (Profiles)
To evaluate token efficiency configs (DCP, model routing, compaction settings), fixtures can declare profiles:
The eval runner executes each fixture against each profile, producing separate
runs.dbrows with distinctprofile_idvalues. The summary report shows a comparison matrix.Rollout Plan
Phase 1: Foundation (eval-infra repo)
unbound-force/eval-infrarepoopencode-driver.tsandanswers.tsfrom scriptedtoken-capture.ts(parsestep_finishevents)store.ts(runs.dbschema + operations)fingerprint.ts(row identity hashing)run-eval.tsCLI entrypointreusable_opencode_eval.ymlworkflow with GCP WIF authPhase 2: First Consumer
.eval/directory tounbound-force/unbound-force/opsx-propose,/triage-issue,/review-prci_eval.ymlcaller workflowbaseline.jsonfrom first successful runsPhase 3: LLM Judge
judge.ts(rubric parsing, LLM judge call, score aggregation)runs.dbrows and baseline comparisonPhase 4: Expansion
/unleash,/finale)Open Questions
Should
eval-infraalso host the sharedanswers.tspatterns, or should each repo bring its own? Recommendation: shared base patterns in eval-infra, with per-fixtureanswers.yamloverrides in consuming repos.How often should eval run? On every PR that touches
.opencode/or.eval/? On a schedule? Both? Recommendation: PR-triggered for regression detection, weekly scheduled for drift monitoring.Should
runs.dbbe committed or artifact-only? Committing it enables historical queries locally but creates merge conflicts. Artifact-only means history lives in GitHub Actions artifacts (90-day retention). Recommendation: artifact-only, with a periodic job that consolidates into a long-term store if needed.Multi-judge from the start or single-judge first? Recommendation: single judge initially, add multi-judge when we have enough data to calibrate disagreement thresholds.
Where should the
scriptedbatch-fix code live long-term? It currently exists only locally. The driver portions should move into eval-infra; the batch-fix orchestration (issue iteration, PR creation) is a separate concern and could stay in its own repo or become a separate workflow.All reactions