2/3 Add the redesigned VeRO system (vero/)

.gitignore

@@ -1,5 +1,6 @@
 # Python-generated files
 __pycache__/
+.pytest_cache/
 *.py[oc]
 build/
 dist/
@@ -9,3 +10,11 @@ wheels/
 # Virtual environments
 .venv
 .env*
+
+# Run secrets loaded via `vero harbor run --env-file` (keep only *.example)
+secrets.env
+*.secrets.env
+!*.example
+
+# Local benchmark scoping notes (paper planning; not tracked)
+harness-engineering-bench/benchmark-scoping.md

README.md

@@ -1,122 +1,86 @@
-# VeRO: Versioning Rewards and Observations
+# VeRO: a harness for agents to optimize programs, text, and agents
+> The code for the original VeRO paper is archived in [`legacy/`](legacy/).
 
 [![Paper](https://img.shields.io/badge/arXiv-2602.22480-b31b1b.svg)](https://arxiv.org/abs/2602.22480)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-
-VeRO is an evaluation harness for using coding agents to optimize LLM-based agents and workflows. It treats agent code as a versioned artifact — making changes, evaluating results, and hill-climbing toward better performance using git version control.
-
-> **Paper**: [VeRO: An Evaluation Harness for Agents to Optimize Agents](https://arxiv.org/abs/2602.22480)
-
-## Repository Structure
-
-```
-vero/
-├── vero/               # Core library (scale-vero)
-├── vero-agents/        # Agent implementations (benchmarking targets)
-├── vero-benchmarking/  # Benchmarking scripts and analysis
-└── LICENSE
-```
-
-### [vero/](vero/)
-
-The core optimization framework. Provides:
-
-- **Policy** — orchestrates the optimization loop (agent + evaluator + git)
-- **Agents** — VeroAgent (OpenAI Agents SDK) and ClaudeCodeAgent (Claude Agent SDK)
-- **Evaluator** — runs task evaluations in isolated subprocess environments
-- **Tools** — MCP-based tools for agents (bash, file I/O, experiment runner, dataset viewer, etc.)
-- **Traces** — session analysis and LLM-based trace interpretation
-
-```bash
-cd vero && uv sync --extra optimize
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/)
+
+VeRO gives a coding agent something to edit, an evaluation boundary, and durable
+memory of every candidate it tried. The target is anything you can put under Git
+and score — a **program** (a single function up to a whole codebase), **text**
+(a prompt, spec, or config), or an **agent** (its scaffold, tools, and prompts).
+Agents are programs, but not everyone reads "program" that way, so VeRO names
+them explicitly: it was introduced to optimize agents and generalizes the same
+version / evaluate / select loop to any Git-versioned artifact.
+
+```mermaid
+flowchart LR
+    S["Strategy<br/>proposes ideas"] --> P["Candidate producers<br/>edit isolated workspaces"]
+    P --> E["Evaluation backends<br/>score versioned reports"]
+    E --> Sel["Selection keeps the<br/>best feasible candidate"]
+    Sel -->|"next round"| S
 ```
 
-See [vero/README.md](vero/README.md) for full documentation.
-
-### [vero-agents/](vero-agents/)
+The target and evaluator do not need to be Python. External evaluators and
+candidate producers connect through command protocols; Python benchmarks can
+use the optional, optimizer-independent `scale-vero-tasks` package.
 
-Agent implementations used as optimization targets:
+Targets may live locally or in an isolated sandbox. VeRO keeps optimization
+state and experiment tracking on the host while running Git worktrees, producer
+commands, builds, and evaluation commands in the target sandbox. The core guide
+includes a no-bind-mount `DockerSandbox` example.
 
-| Agent | Description |
-|-------|-------------|
-| **generic-agent** | General-purpose agent for MATH, GPQA, GAIA, GSM8K, etc. |
-| **web_search_agent** | Web search agent for SimpleQA, Facts Search |
-| **KIRA** | Terminal task agent for Terminal Bench 2.0 |
-| **tau-bench** | Customer service tool-use agent |
-| **pharma_summarizer** | Document summarization agent |
+## Repository layout
 
-See [vero-agents/README.md](vero-agents/README.md) for details.
+| Directory | Purpose |
+| --- | --- |
+| [`vero/`](vero/) | The `scale-vero` optimization kernel, runtime, CLI, and coding-agent adapters |
+| [`vero-tasks/`](vero-tasks/) | Narrow Python task types and schema-v1 evaluation runner |
+| [`harness-engineering-bench/`](harness-engineering-bench/) | Harbor-native target programs and end-to-end optimization benchmarks |
 
-### [vero-benchmarking/](vero-benchmarking/)
+Start with the [generic C matrix-multiplication quickstart](vero/examples/c-matmul/),
+try the [26-circle packing benchmark](vero/examples/circle-packing/), or read the
+[core guide](vero/README.md). The C example demonstrates the language-neutral
+command protocol without model credentials. Circle packing is a substantive
+coding-agent benchmark with exact geometry checks and inspectable search
+artifacts.
 
-Scripts and infrastructure for running optimization experiments:
+For a full agent-optimization example, the [GAIA benchmark](harness-engineering-bench/gaia/)
+pairs a tool-using GPT-5.4 mini target with Harbor's canonical GAIA verifier and
+an immutable 20% / 40% / 40% development, validation, and test split.
 
 ```bash
-cd vero-benchmarking && uv sync --all-extras
-
-# Run an optimization experiment
-uv run python scripts/run_benchmark.py --scaffold claude-code-vmf --model sonnet --task math
-
-# Build datasets
-./scripts/build_datasets.sh
+cd vero
+uv sync --all-extras
+uv run vero --help
 ```
 
-See [vero-benchmarking/README.md](vero-benchmarking/README.md) for full documentation.
-
-## Quick Start
+## Paper reproduction
 
-### Prerequisites
+VeRO was introduced in [*VeRO: A Harness for Agents to Optimize
+Agents*](https://arxiv.org/abs/2602.22480), accepted at ICML 2026. The current
+library generalizes that version/evaluate/select loop from agents to programs.
 
-- Python 3.11+
-- [uv](https://docs.astral.sh/uv/getting-started/installation/)
-- Git
-- Access to an LLM provider (via LiteLLM, OpenAI, Anthropic, etc.)
-
-### Install
+The exact paper implementation is frozen separately from the v0.5 redesign:
 
 ```bash
-git clone <repo-url> && cd vero
-
-# Install core library
-cd vero && uv sync --extra optimize
-
-# Install benchmarking tools
-cd ../vero-benchmarking && uv sync --all-extras
+git checkout paper-v1
 ```
 
-### Run Your First Optimization
-
-```python
-from agents import Agent as OAIAgent
-from vero.policy import Policy
-from vero.agents.vero import VeroAgent
-
-policy = Policy(
-    project_path="/path/to/my-agent",
-    dataset="/path/to/my-dataset",
-    agent=VeroAgent(
-        oai_agent=OAIAgent(name="VeroAgent", model="anthropic/claude-sonnet-4-5-20250929"),
-    ),
-    task="main",
-    train_budget=10,
-    max_turns=200,
-)
-
-best = await policy.run()
-print(f"Best commit: {best.commit}, score: {best.score}")
-```
+Use the `paper/v1` branch or `paper-v1` tag for reproduction. Development of
+the generic program optimizer continues on the `v0.5` branch. The frozen ref
+also preserves the paper-era `vero-agents` and `vero-benchmarking` directories;
+their Harbor-native replacement on `v0.5` is `harness-engineering-bench`.
 
 ## Citation
 
 ```bibtex
 @article{ursekar2026vero,
-  title={VeRO: An Evaluation Harness for Agents to Optimize Agents},
+  title={VeRO: A Harness for Agents to Optimize Agents},
   author={Ursekar, Varun and Shanker, Apaar and Chatrath, Veronica and Xue, Yuan (Emily) and Denton, Sam},
   journal={arXiv preprint arXiv:2602.22480},
   year={2026}
 }
 ```
 
-## License
-
-[MIT](LICENSE)
+VeRO is licensed under the [MIT License](LICENSE).

runs/.gitignore

@@ -0,0 +1,3 @@
+# Local GAIA/harbor run outputs — inspectable, not committed.
+*
+!.gitignore

vero-tasks/README.md

@@ -0,0 +1,35 @@
+# scale-vero-tasks
+
+`scale-vero-tasks` is the optional Python task protocol used by VeRO benchmark
+targets. It contains no optimizer, workspace, session, dataset-store, or
+experiment-database code.
+
+```python
+from vero_tasks import TaskOutput, TaskResult, create_task
+
+task = create_task("exact_match")
+
+@task.inference()
+async def infer(case, context):
+    return TaskOutput(output=case["question"].upper())
+
+@task.evaluation()
+async def evaluate(case, output, context):
+    return TaskResult.from_task_output(
+        output,
+        score=float(output.output == case["answer"]),
+    )
+```
+
+The runner accepts a VeRO command-evaluation request and an external JSON/JSONL
+case file, imports a task module, and writes a schema-v1 evaluation report. The
+standard VeRO adapter runs this package in a trusted evaluator project while
+overlaying the candidate package as an editable dependency. This keeps Python
+benchmark ergonomics separate from both the target program and VeRO's
+language-neutral evaluation kernel.
+
+The runner applies VeRO's retry policy independently to each non-batch
+inference and evaluation call. Provider rate limits, configured HTTP status
+codes or messages, and timeouts can be retried with bounded exponential
+backoff. Earlier failed attempts are retained in the canonical case error list;
+they are non-terminal when a later attempt succeeds.

vero-tasks/examples/matmul-eval/pyproject.toml

@@ -0,0 +1,16 @@
+[project]
+name = "matmul-eval"
+version = "0.1.0"
+description = "Evaluation task for the matmul kernel — measures correctness and speed"
+requires-python = ">=3.11"
+dependencies = ["scale-vero-tasks>=0.2.0"]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/matmul_eval"]
+
+[tool.uv.sources]
+scale-vero-tasks = { path = "../..", editable = true }

vero-tasks/examples/matmul-eval/src/matmul_eval/matmul_task.py

@@ -0,0 +1,70 @@
+"""Matrix multiply evaluation task.
+
+Measures correctness and execution speed of a matmul kernel.
+Score = average time per multiply (lower is better).
+Incorrect results get a penalty score of 999999.0.
+"""
+
+import time
+
+from vero_tasks import (
+    TaskContext,
+    TaskOutput,
+    TaskParameters,
+    TaskResult,
+    create_task,
+)
+
+
+class MatmulParameters(TaskParameters):
+    n_repeats: int = 100
+
+
+matmul_task = create_task("matmul", task_parameters_type=MatmulParameters)
+
+
+@matmul_task.inference()
+async def run_inference(task: dict, context: TaskContext) -> TaskOutput:
+    from matmul_kernel import multiply
+
+    a = task["matrix_a"]
+    b = task["matrix_b"]
+
+    parameters = context.parse_task_params(MatmulParameters)
+
+    # Warmup
+    multiply(a, b)
+
+    # Timed runs (like timeit)
+    start = time.perf_counter()
+    for _ in range(parameters.n_repeats):
+        result = multiply(a, b)
+    elapsed_ms = (time.perf_counter() - start) / parameters.n_repeats * 1000
+
+    return TaskOutput(output={"result": result, "time_ms": elapsed_ms})
+
+
+@matmul_task.evaluation()
+async def run_evaluation(
+    task: dict, output: TaskOutput, context: TaskContext
+) -> TaskResult:
+    expected = task["expected"]
+    actual = output.output["result"]
+    time_ms = output.output["time_ms"]
+
+    # Check correctness (within floating point tolerance)
+    correct = all(
+        abs(a_val - e_val) < 1e-6
+        for a_row, e_row in zip(actual, expected)
+        for a_val, e_val in zip(a_row, e_row)
+    )
+
+    # Score = time if correct, penalty if wrong (lower is better)
+    score = time_ms if correct else 999999.0
+
+    return TaskResult.from_task_output(
+        output,
+        score=score,
+        metrics={"time_ms": time_ms, "correct": 1.0 if correct else 0.0},
+        feedback=f"{'Correct' if correct else 'Wrong'}. Time: {time_ms:.2f}ms",
+    )