feat(tasks): PR complexity reviewer as an assess_brief specialization (rubric + reference executor)

[legreffier]

Spun out of #852.

Insight

A "PR complexity reviewer" doesn't need a new task type. assess_brief already exists in BUILT_IN_TASK_TYPES with output_kind: 'judgment' and a rubric-driven scoring shape. The PR-review use case is just assess_brief instantiated with:

• input.brief — the PR diff / description / acceptance criteria
• input.references[] — pointers to the PR (URL, commit SHA, optional snapshot CID)
• input.criteriaCid — a published "PR-complexity rubric" CID
• An executor that wraps pi-review (or gh pr diff + LLM) against that rubric

This validates the runtime's genericity: same task type, new rubric, new executor — no schema changes. It's also the proof point #852 wanted before declaring the runtime ready for second-domain use.

Deliverables

1. Rubric authoring (packs or a fresh rubrics directory):

   • Criteria: cognitive_load, blast_radius, test_coverage_delta, security_surface_touched, style_compliance, optionally migration_safety.
   • Each criterion: { id, weight, prompt, rangeMin, rangeMax } matching the existing rubric schema used by pack-fidelity-v2.
   • Pin to a CID so reproducibility is content-addressed.

2. Reference executor in libs/pi-extension or a new libs/pr-review-extension:

   • Polls /tasks?taskType=assess_brief&criteriaCid=<pr-complexity-rubric-cid>.
   • Resolves the PR via references[] (GitHub API + agent token via the legreffier rule).
   • Runs pi-review (or equivalent) inside Gondolin against the diff.
   • Returns output: { scores: [...], composite, rationale } matching the existing judgment shape.

3. e2e test: against a known GitHub PR (or a fixture diff), assert the executor produces a judgment that matches expected scores within tolerance. Lives in apps/rest-api/e2e/ or a new apps/agent-daemon/e2e/.

4. Docs: short section in docs/agent-runtime.md ("Second concrete task: PR review") explaining the assess_brief + custom-rubric + custom-executor pattern. Use it as the canonical "how to add a new domain" example.

Why not a new task type

Considered and rejected: a review_pr task type. It would duplicate assess_brief's entire schema (output_kind: judgment, rubric reference, scoring shape) just to attach a domain label. That fragments the runtime — every new domain (security review, design review, accessibility review, …) would want its own type. The right axis of variation is rubric + executor, not task type.

Acceptance criteria

• PR-complexity rubric authored, CID-pinned, committed alongside existing rubrics.
• Reference executor wraps a real PR-review path (pi-review or equivalent) against assess_brief.
• e2e test validates one round trip end-to-end (claim → execute → complete with scores).
• Doc explains the pattern as a template for future domain-specific assessments.

Related

• PR-review sidecar eval: next injection-format experiments #836 PR-review sidecar eval (this is the runtime side; PR-review sidecar eval: next injection-format experiments #836 is the eval-format side)
• Proctored evals on the task runtime (supersedes #523) #943 Proctored evals on the task runtime (this PR's executor will eventually be a proctored target)
• pi-extension: honor reporter.cancelSignal in executePiTask (call session.abort) #947 pi-extension: honor reporter.cancelSignal (the executor needs to honor this when wrapping pi)

[legreffier]

Scope revised after looking at what apps/agent-daemon + libs/pi-extension already do

The original plan called for a "reference executor" wrapping pi-review. After auditing the daemon and the existing assess_brief pipeline, there is no new executor needed. The runtime's genericity already covers PR review.

What's already in place

• Daemon (apps/agent-daemon) polls /tasks?taskTypes=..., claims, and dispatches via createPiTaskExecutor. Already configurable to assess_brief.
• Prompt routing (libs/agent-runtime/src/prompts/index.ts) handles assess_brief today, including resolving a target (PR url, branch, commits, diary entries) into the prompt.
• Rubric-driven scoring is part of AssessBriefInput.criteria[]. Each criterion carries id, weight, scoring (llm_judged / boolean), and a description prompt.
• Agent GH token injection inside Gondolin is already wired (libs/pi-extension/src/vm-manager.ts): the VM gets moltnet.json and the GitHub App PEM at start; inside the VM, moltnet github token --credentials ... mints an agent-scoped token. No host/human token leakage — this is exactly the legreffier pattern.

What actually needs to change

Item	Where	Cost
Add api.github.com to allowedHosts so gh pr ... works inside the VM	sandbox.json	1 line
Author the PR-complexity rubric (criteria array)	New file: libs/tasks/src/rubrics/pr-complexity.ts (or JSON next to existing rubrics)	~50 lines of data
Imposer CLI: read gh pr view, build AssessBriefInput with the rubric inline + PR references, POST	New file: tools/src/tasks/assess-pr.ts, near-clone of tools/src/tasks/fulfill-brief.ts	~80 lines
Optional: extend prompt target.kind discriminator so the judge prompt can say "use gh pr diff to read the change" when the target is a PR	libs/agent-runtime/src/prompts/assess-brief.ts	~20 lines, backwards-compatible
e2e against a fixture PR	apps/rest-api/e2e/ or apps/agent-daemon/e2e/	~150 lines

Why not a domain-specific extension

A libs/pr-review-extension would duplicate createPiTaskExecutor to bake in PR-specific tooling. But the daemon already has bash + read tools available; gh is just another binary. Adding a fresh extension to wrap the same pi-coding-agent against the same task type with the same scoring shape is cargo-cult abstraction.

The rubric is the variation point. The executor is generic.

Revised acceptance criteria

• sandbox.json allowedHosts includes api.github.com (and any required CDNs like objects.githubusercontent.com for diff downloads).
• PR-complexity rubric authored as a CID-pinned constant in libs/tasks/src/rubrics/.
• tools/src/tasks/assess-pr.ts mirrors fulfill-brief.ts: takes --pr <number|url>, posts an assess_brief task with the rubric inline.
• assess-brief.ts prompt builder optionally takes target.kind: 'pull_request' and includes a one-liner about reading the diff.
• e2e: imposer + daemon + a known PR → judgment with sensible scores, lands on the attempt's output.
• Doc in docs/agent-runtime.md shows the rubric+imposer pattern as the canonical "add a new domain" template.

This is a much smaller PR than originally scoped, and that's the whole point: the runtime really is generic.

[legreffier]

Discovered while implementing: the daemon does not currently support assess_brief end-to-end

While writing the rubric + imposer per the revised plan, I checked the daemon's executor path. assess_brief is in the registry but doesn't actually run today.

What's missing

libs/agent-runtime/src/prompts/index.ts:64 requires ctx.extras.target to be populated for assess_brief:

case ASSESS_BRIEF_TYPE: {
  // ...
  const target = ctx.extras?.target as ...;
  if (!target) {
    throw new Error(
      'assess_brief prompt requires ctx.extras.target (resolved fulfill_brief summary)',
    );
  }
  // ...
}

The daemon's executePiTask accepts promptExtras but the daemon never populates it. There's no target resolution step. Grepping apps/agent-daemon/ for assess_brief / target / promptExtras returns nothing.

So today: posting an assess_brief task and letting the daemon claim it would fail at prompt-build with the error above. The agent-daemon e2e from #944 only exercises fulfill_brief for that reason.

Implication for this PR's scope

The original revised plan ("rubric + imposer + sandbox tweak, no executor changes") was based on the premise that the daemon already runs assess_brief. It doesn't. So #951 needs to also wire up target resolution.

Two paths

(a) Resolve target from targetTaskId — fetch the referenced task, project its output into target. Works for fulfill_brief judgments. Doesn't work for PR review (no fulfill_brief task to reference).

(b) Resolve target from references[].external — generalize the prompt builder so target can be constructed from a github_pr reference, a github_issue reference, or a fulfill_brief task id. The prompt builder picks based on a discriminator. Each producer of an assess_brief task chooses what it's pointing at via references[].

(b) is the right design. It also retroactively explains why the schema makes targetTaskId required in AssessBriefInput — that decision was rushed; target should really be a discriminated union.

Updated plan for #951

1. Refactor AssessBriefInput to accept a discriminated target field (or relax targetTaskId to optional and derive from references[]). Backwards-compatible: the existing fulfill_brief case keeps working via the same field name.
2. Refactor buildAssessBriefPrompt to dispatch on the target kind (fulfill_brief_summary | pull_request | issue).
3. Wire daemon target resolution: before calling executePiTask, the daemon reads the task's references / target and constructs promptExtras.target. Add a small libs/agent-runtime/src/prompts/resolve-target.ts that does this from a Task, plus deps for fetching a target task when needed.
4. PR-complexity rubric (already written in this branch).
5. Imposer tools/src/tasks/assess-pr.ts (already written in this branch).
6. Sandbox tweak for api.github.com egress.
7. e2e: claim, run against a real fixture PR, assert score shape.

This is back to the original "build PR reviewer as second concrete task type" scope from #852, but the second task type isn't review_pr — it's assess_brief actually working end-to-end for the first time with PR review as the proving use case.

Asking before I keep going: should I land this as one PR, or split it into:

• #951a "make assess_brief actually work end-to-end" — the target resolution refactor + daemon wiring (proves the runtime is generic in fact, not just on paper).
• #951b "PR-complexity reviewer" — the rubric + imposer + sandbox tweak.

#951a is the prerequisite. I'd prefer to split because the target-resolution refactor is reviewable on its own merits and unblocks any future assess_brief use case (security review, design review, etc.).