Evaluate Goal based criteria for dev-lead

[don-petry]

Some harnesses like Claude support the use of defined criteria that just be achieved for the agent to complete. This causes the agent to continue working until conditions are met thereby increasing the success rate of the session.

Research, evaluate, analyze and recommend if and where this could be used.

https://code.claude.com/docs/en/goal

https://www.langchain.com/blog/introducing-rubrics-for-deepagents

[github-actions[bot]]

Enhancement: Goal-Based Criteria — Specific Workflows & Failure Mode Analysis

Sharpened problem & goal
The proposal is to research and evaluate goal-based criteria (Claude Code's goal system, LangChain's rubrics for DeepAgents) for use in the org's agent harnesses. Goal-based criteria cause an agent to loop until specified conditions are met, increasing task completion rates at the cost of potentially higher token usage per run. The key question the research should answer: for which specific org workflows is the current "one-shot, done" behavior causing measurable quality loss — and would goal-based looping fix it, or would it just add token spend?

Context
The org's existing agent workflows have different completion patterns that inform where goals apply:

• dev-lead-reusable.yml already has dev-lead-retry.yml + auto-rebase-retry.yml for failure recovery at the workflow level; goal-based criteria would operate at the agent session level — retry until CI green, not just retry on workflow infrastructure failure
• scripts/dev-lead-lint.sh — already checks for linting success post-fix; this IS a goal criterion applied manually. The proposal would formalize this pattern
• scripts/dev-lead-fix-ci.sh — "fix CI failures" is the canonical use case for a CI_MUST_PASS goal criterion; it is the highest-value pilot target
• Discussion RFC: Deterministic pre-agentic review gate to cut token burn (draft-until-green) #743 (pre-agentic gate) and Discussion 💡 Adversarial Plan-Critic + Structural Gates for the Initiative-Planner (Bob) #593 (plan-critic) both define acceptance criteria; goal-based criteria could enforce them automatically instead of relying on post-hoc checking

Token cost implication: a goal-based loop that runs 3× to achieve a CI-green outcome may still be cheaper than a human engineer's time, but more expensive than a single successful agent run. The evaluation must include a cost-per-success comparison, not just a success-rate comparison.

Impact / Effort

• Impact: MEDIUM — dev-lead already has retry logic; goal-based criteria improve first-attempt quality for well-defined tasks (CI-gated fixes), not exploratory tasks; the win is in reducing the manual retry loop
• Effort: S for research + recommendation doc; M for piloting in scripts/dev-lead-fix-ci.sh with a CI-green goal criterion

Suggested acceptance criteria

• A research doc (docs/goal-criteria-evaluation.md) covers: Claude Code goal mechanics (verification method, token implications of looping), LangChain rubrics, and at least one alternative (e.g., DSPy assertions); includes a comparison table
• The doc identifies the top 3 org workflows by observed failure rate / manual-retry frequency where goal criteria would most help — grounded in real data (e.g., dev-lead CI-fix success rate from scripts/evals/run-eval.sh)
• A pilot implements a CI_MUST_PASS goal criterion in scripts/dev-lead-fix-ci.sh and runs for 2 weeks, reporting: task completion rate, token cost per successful fix (vs. current one-shot baseline), and false-positive rate (declared success with CI still failing)
• The recommendation includes an explicit list of workflows that should not use goal criteria (e.g., open-ended PR reviews where "done" is inherently subjective) with reasoning
• Token cost per successful completion (goal loop total) is compared to current cost via token-report.yml — the goal is to show goal criteria are net-positive on cost-per-success, not just completion rate