[FEAT][Implement Planner Generator Evaluator Multi-Agent Harness from Anthropic]

[kyegomez]

Implement a domain-agnostic three-agent orchestration harness (Planner, Generator, Evaluator) inspired by the GAN-style architecture described in Anthropic's harness design research. The harness coordinates long-running autonomous tasks from a short natural-language prompt, using an iterative generate-evaluate feedback loop to converge on high-quality output across any domain.

All three agents communicate through a single shared file on disk.

The implementation should live in a new PGEHarness class with a public run() method as the primary entry point, orchestrating three internal agent classes: Planner, Generator, and Evaluator.

---

Background and Motivation

Naive single-agent implementations of long-running tasks exhibit two well-documented failure modes:

1. Context degradation: As the context window fills, the agent loses coherence or begins wrapping up prematurely ("context anxiety"). Context resets (clearing the window and handing off structured state) address this more effectively than in-place compaction alone.
2. Self-evaluation bias: When asked to evaluate its own output, an agent skews positive and overlooks real defects. This holds for both subjective criteria (quality, taste) and verifiable outcomes (correctness, completeness).

Separating generation from evaluation into distinct agents, and adding an upstream planning step, addresses both issues. The evaluator can be independently tuned for skepticism, and its concrete feedback gives the generator something actionable to iterate against.

---

Architecture Overview

The harness consists of three cooperating agents orchestrated sequentially within a single run. All inter-agent communication flows through one shared file.

                        +------------------------------------------+
                        |            shared_state.md                |
                        |  (single file, all agents read/write)    |
                        +------------------------------------------+
                              |              |              |
                           reads          reads          reads
                           writes         writes         writes
                              |              |              |
                              v              v              v
  prompt --> [ Planner ] --> [ Generator ] <-----> [ Evaluator ]
                                    ^                    |
                                    |   feedback loop    |
                                    +--------------------+

The Shared State File

All agent communication is mediated through a single file (shared_state.md or equivalent). Each agent appends structured sections to this file and reads sections written by other agents. The file serves as the complete, append-only record of the task: the plan, the work log, contracts, evaluation results, and feedback. No direct in-memory communication occurs between agents.

The file uses clearly delimited sections (e.g., headers or markers) so each agent can locate the information it needs. The harness is responsible for directing agents to read from and write to the appropriate sections.

Agent Roles

1. Planner

• Input: A short user prompt (1-4 sentences).
• Output: A structured plan appended to the shared state file.
• Responsibilities:
  • Expand the prompt into an ambitious, detailed plan.
  • Define the scope, deliverables, and high-level approach.
  • Stay at the strategic level; avoid specifying granular execution details that could cascade errors downstream.
  • Define the evaluation criteria and quality dimensions that the Evaluator will use to grade the Generator's work.

2. Generator

• Input: The plan and evaluation criteria (from the shared state file), and on subsequent iterations, evaluation feedback.
• Output: The produced work product (written to disk or working directory), plus a work log appended to the shared state file.
• Responsibilities:
  • Execute the plan step by step, producing concrete output.
  • Before each step, negotiate a "step contract" with the Evaluator via the shared state file: a mutual agreement on what "done" looks like for that step, including testable acceptance criteria.
  • On receiving evaluation feedback, either refine the current output or pivot to a different approach depending on score trajectory.

3. Evaluator

• Input: The Generator's output, the step contract, and the evaluation criteria (all read from the shared state file).
• Output: A structured evaluation appended to the shared state file, including per-criterion scores, a pass/fail determination, and actionable feedback.
• Responsibilities:
  • Review and test the Generator's output against the agreed-upon contract and criteria.
  • Enforce hard score thresholds; fail the step if any criterion falls below its configured minimum.
  • Provide specific, actionable findings rather than vague assessments.
  • The method of review is domain-dependent and should be configurable (e.g., browser automation for web apps, file inspection for documents, execution for code).

Quality Criteria

Evaluation criteria are not hardcoded. The Planner defines them as part of the plan based on the domain and prompt. The harness provides a default structure that the Planner populates:

Field	Description
Criterion name	A short label for the quality dimension.
Weight	Relative importance (high, standard, low).
Description	What the criterion measures and what good/bad looks like.
Threshold	Minimum passing score.

This allows the same harness to evaluate a research report on analytical depth and source quality, a codebase on correctness and maintainability, a design on cohesion and originality, or any other domain-appropriate dimensions.

Orchestration Flow

run(prompt) ->
  1. Planner reads prompt, appends plan + evaluation criteria to shared state file
  2. For each step (or until plan is complete):
     a. Generator proposes a step contract (appends to shared state file)
     b. Evaluator reviews contract, negotiates amendments (appends to shared state file)
     c. Generator executes the step, produces output, appends work log
     d. Evaluator reviews output, appends evaluation with scores and findings
     e. If any criterion below threshold: Generator reads feedback, repeats from (c)
     f. If all criteria pass: proceed to next step
  3. Return final output path and summary

---

Proposed Implementation

Class: PlannerGeneratorEvaluator

class PlannerGeneratorEvaluator:
    def __init__(self, config: HarnessConfig):
        """
        Parameters:
            config: HarnessConfig containing:
                - model: str (model identifier)
                - max_steps: int (upper bound on plan steps)
                - max_retries_per_step: int (max evaluation failures before advancing)
                - working_directory: Path (where output is produced)
                - shared_state_path: Path (the single communication file)
                - default_thresholds: dict[str, float] (fallback score thresholds)
        """

    def run(self, prompt: str) -> HarnessResult:
        """
        Execute the full harness pipeline from a short user prompt to a
        completed output.

        Parameters:
            prompt: A short natural-language description of the desired task.

        Returns:
            HarnessResult containing:
                - output_path: Path to the final deliverable
                - plan: The generated plan
                - step_logs: List of per-step metadata (contract, scores, retries)
                - total_duration: Wall-clock time
                - total_cost: Estimated token cost
        """

Supporting Types

• Planner: Agent class responsible for plan and criteria generation.
• Generator: Agent class responsible for execution and output production.
• Evaluator: Agent class responsible for review and scoring.
• HarnessConfig: All tunable parameters (model, thresholds, max iterations, directory paths).
• HarnessResult: Final output container with deliverable path, plan, step logs, duration, and cost.
• StepContract: Structured representation of what the Generator will produce and how the Evaluator will verify it for a given step.
• EvaluationReport: Per-criterion scores, pass/fail status, and itemized findings.

Agent Implementations

Each agent should be encapsulated as its own internal class (Planner, Generator, Evaluator), invoked by the harness:

• Planner.run(prompt: str, shared_state_path: Path) -> None -- appends plan and criteria to the shared state file.
• Generator.run(shared_state_path: Path) -> Path -- reads plan/feedback from shared state, produces output, appends work log. Returns path to output.
• Evaluator.run(shared_state_path: Path) -> EvaluationReport -- reads output and contract from shared state, appends evaluation. Returns structured report.
• PGEHarness._negotiate_contract(step_number: int) -> StepContract -- manages the negotiation loop between Generator and Evaluator via the shared state file.

---

References

• Source: "Harness design for long-running application development" (Anthropic Engineering, March 2026)
• Related concept: GAN-inspired generator-evaluator feedback loop
• Related concept: Context resets vs. compaction for long-running agent sessions

https://www.anthropic.com/engineering/harness-design-long-running-apps