2/3 Add the redesigned VeRO system (vero/)#45
Open
varunursekar wants to merge 31 commits into
Open
Conversation
Introduce the v0.5 codebase: vero/ (core + harbor eval sidecar, build compiler, runtime/observability), vero-tasks/, and runs/. Root README points to legacy/ for the original-paper code.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…back #1) Addresses the 'baseline floor fails open' regression: - baseline_floor now defaults to false. The floor gates a ship on a validation comparison while the reward is on the (possibly differently-distributed) target, so it is opt-in. - Pin the fixed seed's score to avoid re-scoring it every run (reproducibility + one fewer eval): VerificationTarget.baseline_reward (per-target, post scale/offset) and VerificationSelection .baseline_selection_score (for the floor). Plumbed through the build config + compiler. - Fail safe when the floor is on, unpinned, and the seed re-score can't be measured: ship nothing (NoCandidateError -> shipped=false + infra error) instead of shipping the best candidate unverified (the fail-open bug). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#1) Add an admin-only way to generate the number to pin: CanonicalVerifier .measure_baseline(replicates) admin-scores the fixed seed N times on the selection partition and each target and returns per-key mean/stddev. Exposed as POST /score/baseline and 'vero harbor score-baseline --replicates N'. Reuses the existing admin scoring machinery. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…atten The benchmark-config tests read harness-engineering-bench/ (a separate branch in the stacked split), so skip them when it isn't checked out. Update the paths for the candidates/ -> top-level flatten (all benchmarks are now top-level siblings of gaia/). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
min_aggregate_cases defaulted to 1 — a vacuous floor that let an agent read held-out validation labels one case at a time via single-case aggregate evals. Default to 5 in both the sidecar policy and the build AgentAccessSpec so a build that omits it is safe rather than unfloored (shipped builds already set 5 explicitly). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-enabling harness isolation broke the eval: 'uv run' as the unprivileged harness got a fresh empty uv cache and couldn't resolve the candidate package from it (No module named 'gaia_agent'), with no network to download at eval time. Pre-warm /home/harness/.cache/uv from the build (root) cache in the sidecar image, matching the UV_CACHE_DIR the backend sets for the harness user. Also fixes a test regression from feedback #2: with the k-anonymity floor defaulting to 5, the instruction now reads 'must include at least 5 cases' rather than 'arbitrary subsets'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ack #8) The mean-of-k recorded only score/n_attempts/n_scored, so an outage-diluted cell (dead attempts zero-filled) was indistinguishable from a genuinely clean low cell. Add n_dead_infra (dead-to-infrastructure attempts, classified via the existing error taxonomy) and n_clean (the rest), so the split is visible in the live metrics rather than only reconstructable from raw trial records. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- optimizer.run: validate self.max_proposals >= 0 before the proposal_limit guard, so the specific 'must be non-negative' error is reachable (was a dead branch that always tripped the generic message first). - compiler._safe_extract_tar: extractall(filter='data') to strip device files / setuid bits and neutralize unsafe links, matching extract_harbor_session_archive (also silences the py3.14 deprecation). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cold-start walkthrough of the harbor path: the three-container trust boundary, a run end-to-end, the evaluation core, disclosure/selection, the five security mechanisms, and a suggested outside-in review order keyed to the source files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update harbor_requirement pin from harbor[modal]==0.18.0 to ==0.20.0 across the README and harbor tests. Remove the PYTHONPATH injection in the harness eval launch: it was a no-op (uv strips PYTHONPATH from the child env) and did not fix the "No module named <agent>" failure under the dropped uid, so it and its test assertion are gone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The candidate is checked out at <mktemp>/repository; `mktemp -d` makes that parent 0700 root. The eval launch chowned the repository (and the staging tree) to the harness user but not the mktemp parent, so once the process dropped to harness it could not traverse into its own workspace to resolve the editable candidate package's absolute path — every eval (baseline included) failed with "No module named <agent>" while harbor itself still loaded from the harness-owned cache. Grant the harness user traversal on that parent (chmod o+x) alongside the existing chowns. o+x suffices because run_as drops to the user's own uid+gid, so it is "other" relative to the root-owned parent. The dir holds only candidate code, so widening traversal leaks no trusted data. Reproduced and fixed against the real compiled sidecar image with a faithful user+group+extra_groups privilege drop: 0700 -> "No module named gaia_agent"; 0701 -> import resolves. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a reachability probe: after chowning/granting traversal, run `test -r <project_path>` as the dropped user before launching harbor. Readability requires traversing every ancestor, so any provisioning gap (a missing chown, a non-traversable parent) is caught here with a clear message instead of resurfacing as a cryptic "No module named <agent>" several retries downstream — which is exactly how the checkout-parent bug hid. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the harness provisioning commands and reachability probe into vero.harbor.isolation as a single source of truth, and have the eval launch use them, so the runtime and the test can't drift. Add tests/test_v05_harbor_isolation_container.py: inside a throwaway Linux container it loads the real sandbox.py + isolation.py standalone (stdlib-only, no install) and asserts, against the real uid/gid drop and the real provisioning commands, that (1) a checkout under a 0700 mktemp parent is unreachable to the dropped user when only the leaf is chowned (the regression sentinel for the shipped bug), (2) it becomes reachable after harness_grant_commands and the probe agrees, and (3) trusted root-only state stays unreadable to the dropped user. Skips when no docker daemon is available. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents don't know what "vero" is; every name they meet should say what it holds. The workspace context directory is now .evals/ with results/ (was evaluations/), tasks/ (was cases/), and plan.json (was evaluations.json). Host-side paths (~/.vero, ./.vero/session) are operator-facing and unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One well-named entry point over the .evals/ context — the new avatar of the legacy VeroAgent's three structured tools. Runner subcommands (run/status/ result/submit) delegate to the sidecar client lazily; the viewers (list/show/ cases/trace/diff/plan/tasks) are unprivileged stdlib readers of the disclosure- projected tree, with bounded, paginated, metadata-first output. The compiler bakes skills/evals/SKILL.md into the optimizer workspace by default (include_evals_skill), and the instruction template now teaches `evals`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
min_aggregate_cases moves onto the core EvaluationAccessPolicy (omitted resolves to 5 under aggregate disclosure, 1 otherwise) and the engine refuses agent-chosen aggregate subsets below the floor after cost resolution, so the protection holds on every path — programmatic API and local runs included — not just compiled harbor deployments. The sidecar keeps its earlier check as defense-in-depth and now passes the floor through its explicit authorization. AgentAccessSpec.to_access_policy() is the single typed translation from build spec to runtime policy; the compiler emits it nested in serve.json and SidecarEvaluationPolicy holds it as `access`, replacing the hand-rolled flat keys (disclosure/agent_evaluable/min_aggregate_cases/expose_case_resources) that had to be kept in sync across three representations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
varunursekar
force-pushed
the
pr2-add-vero
branch
from
July 24, 2026 03:08
71860e0 to
3fdcf90
Compare
The sidecar's context writer predates the .evals rename and had its own literals: exposed task resources landed in .evals/cases/ (README, skill, and instruction docs all say tasks/) and plan.json was never written in the harbor topology, breaking the documented 'evals plan' first step. Caught live in the first gaia run of the new stack.
AgentContextDirectory now owns write_case_resources and write_evaluation_plan; both WorkspaceContextManager and the sidecar feed it ContextPlanEntry rows (policy, evaluation set, resolved budget), so the tree layout can no longer drift between topologies. Top-level names (results/, tasks/, candidates/, plan.json) are module constants. Behavior note: case-resource exposure now keys on expose_case_resources alone (which model validation already ties to agent_visible); the sidecar previously also required agent_can_evaluate.
…ecar The gateway now keeps a size-rotated JSONL log of every request it proxies or denies (scope, attribution, model, status, latency, tokens, and head+tail-truncated request/response bodies; SSE responses are captured from the chunks already tapped for usage). On by default in compiled tasks at /state/inference/requests with a 16KB per-body cap (inference_gateway.log_requests / request_log_body_bytes). The trusted sidecar - the only credential holder - mirrors the gateway state into the existing W&B run: a poller logs per-scope usage series (live optimizer producer-scope burn included) every 30s and ships rotated log files as artifacts, sharing the sink step counter. At finalize the gateway ledger and request log are copied into the session's artifacts so /session/export preserves every request-response, W&B or not.
The loader resolves an existing local task_source to an absolute path, while the committed partition manifest records it relative to itself; compare resolved locations before rejecting. Registry sources still compare literally.
shehabyasser-scale
approved these changes
Jul 24, 2026
Collaborator
|
Really nice split. Breaking #41 into relocate-legacy / add-system / add-benchmarks makes it reviewable again (Greptile passes each now), keeps the old stack intact under Two things I'd flagged on the earlier v0.5 snapshot are already handled here, thanks for that:
One integrity concern I'd want resolved before this is relied on for competitive selection: Infra-classified cases are excluded from the aggregate rather than scored at If a candidate-attributable failure can reach that path (a candidate that raises Could you confirm whether the routing prevents a candidate-raised exception from being classified transient-infra? If it doesn't, I think the fix is:
One more (high), in Smaller, non-blocking follow-ups:
Happy to pair on the aggregate/retry piece if that's useful. |
The gateway now tracks provider cache reads (cached_input_tokens) per scope and attribution. Each evaluation report carries its own inference_requests/input/cached/output/total_tokens (from the gateway ledger attribution keyed by evaluation id) plus per-case wall_seconds and mean/max_case_wall_seconds from the Harbor trial timestamps. finalize.json echoes the scoring evaluations' full metrics as reward_metrics/baseline_reward_metrics; reward.json is untouched and the score metric remains the sole objective input. Budget-blind mode (disclose_budget: false) redacts inference_* metrics from agent-facing receipts and context projections; latency stays visible and trusted records keep everything.
…threading Part A (always on): each case report carries agent-self-reported token counts from harbor's agent_result (agent_reported_input/cached/output_ tokens) plus report-level sums, distinct from the trusted gateway-metered inference_* metrics; budget-blind mode redacts both prefixes, now also at the per-case level. Part B (experimental, request_log.attribution, default off): the gateway stamps each request-log record with a provider-agnostic conversation thread_id - stateful responses chains inherit via previous_response_id, stateless surfaces (chat completions, Anthropic Messages) group by a first-user-message digest that ignores the system prompt. The attributor is constructed only when flagged, every method is exception-swallowing, and state is FIFO-bounded, so it can never affect proxying. Also tracks cached_input_tokens per scope/attribution.
…o build litellm/pyo3
The v0.5 package (vero/pyproject.toml) declares requires-python = ">=3.11",
which is too permissive. `uv run` picks the newest installed interpreter
(e.g. Python 3.14). The transitive dependency litellm 1.92.0 (pulled via
scale-vero[optimize] -> openai-agents[litellm]) has a Rust/PyO3 core that
fails to build on Python 3.14:
error: the configured Python interpreter version (3.14) is newer than
PyO3's maximum supported version (3.13)
Repro on a machine whose default uv-managed Python is 3.14:
cd vero
uv run vero harbor run ...
uv resolves to Python 3.14 (satisfies >=3.11), then the litellm build fails
with the PyO3 error above. Pinning to <3.14 forces uv to select 3.11-3.13,
matching PyO3's max supported version. Change is limited to vero/pyproject.toml.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a `agent_env` build-config field and render it as harbor `--ae KEY=VALUE` on the optimizer's `harbor run`, so env vars reach the optimizer agent's own setup/install exec. Root cause: `run_command` in vero/src/vero/harbor/cli.py builds the optimizer's harbor command from a fixed list + `-m model` + the click `extra` passthrough and never reads any per-run agent env. `extra_harbor_args` was the only env-adjacent knob, but it is compiled solely into the eval sub-run (compiler.py writes it into serve.json, consumed by backend.py), a different `harbor run` with a different agent, so it never reaches the optimizer agent's install shell. Harbor's `--ae/--agent-env` flag populates the agent's extra_env, which harbor injects into the agent's setup/install exec via scoped_exec_env, so rendering `--ae` here is the supported channel. Motivation: there was no way to set e.g. UV_TOOL_BIN_DIR for the optimizer agent's install on a non-root sandbox (so `uv tool install` doesn't try to symlink into root-owned /usr/local/bin). Passing it via extra_harbor_args silently did nothing. `extra_harbor_args` is intentionally left unchanged to avoid overloading its meaning (eval sub-run args). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SimpleNamespace config stubs in test_v05_harbor_http.py predate #48's agent_env field; run_command now reads config.agent_env, so give the stubs the same defaulted-empty field the real HarborBuildConfig has.
Addresses greptile P1s on #45. In BudgetLedger.reserve/refund a failing durable write (transient OSError) escaped at the shield-drain await and swallowed the run's CancelledError, breaking structured cancellation and leaving a reservation charged on disk. Both drains now catch the write failure and re-raise the originating cancellation, chaining the write error for diagnosis. In the engine, the EvaluationExecutionError handler and the infrastructure-failure refund now wrap record+refund in asyncio.shield like the EvaluationCancelledError handler, so a cancellation racing cleanup cannot interrupt the refund and leak budget.
Addresses the competitive-selection integrity concern on #45. The transient-infra category is detected from the candidate process's own exception type/message, and excluded cases drop out of the aggregate denominator — so a candidate emitting a timeout/connection error on a hard case could inflate its mean, with whole-sub-run retry (default 3) amplifying it under aggregate_attempts: best. For competitive (agent) evaluations a within-trial transient-infra result is now scored at the failure value (the zero-fill invariant), and whole-sub-run infrastructure retry is disabled. Trusted finalization re-scores — run by the operator, not gameable — keep the exclusion and the configured retries. Genuinely harness-produced deadness stays excluded for both: coverage gaps (no trial produced) and gateway-ledger budget/auth (terminating, out of band). Default infrastructure_max_ attempts lowered to 1.
Both specs govern the fair-play boundary but neither said so. Give each a docstring naming its role and fields: AgentAccessSpec is what the optimizer may see and spend while searching a partition, VerificationTargetSpec is how the trusted verifier finally scores the candidate that search picked. Note that min_aggregate_cases is only consulted when disclosure is AGGREGATE; under FULL the agent already sees every per-case result, so the floor is inert there whatever it is set to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EvaluationModel was a BaseModel with extra="forbid" and nothing else, but its name claimed an evaluation-contract scope it never had: WorkspaceOverlaySpec and WandbSpec have nothing to do with evaluation. Worse, the build and runtime layers imported it from vero.evaluation purely to inherit a config flag, which made the evaluation package a dependency of code that otherwise has no reason to touch it. Move the base to vero/models.py as StrictModel, a dependency leaf importing only pydantic, and rebase all 79 subclasses onto it. Fold in the 13 models that hand-rolled the same ConfigDict (Candidate, SessionManifest, RuntimeEvent, the session specs, and others), and drop it from WorkspaceOverlaySpec and HarborBuildConfig, which were setting it redundantly on an already-strict base. Strictness is the point and is unchanged: a typo'd key in a build YAML must fail the build rather than be dropped in silence, leaving a default that quietly widens the optimizer's access. EvaluationModel is removed rather than aliased; it had no references outside src/vero. Test results are identical to before the change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of three stacked PRs splitting #41. Base:
pr1-legacy-relocate.Adds the v0.5 system at the repo root:
vero/(core + harbor eval sidecar, build compiler, runtime/observability),vero-tasks/,runs/. Root README points tolegacy/. Pure adds (197 files) + README/.gitignore.Review after 1/3. 3/3 adds
harness-engineering-bench/on top.🤖 Generated with Claude Code
Greptile Summary
This PR adds the v0.5 VeRO system at the repo root: the
vero/core package (evaluation engine, budget ledger, candidate repository, runtime/W&B observability), the Harbor eval-sidecar and inference gateway, the build compiler, and thevero-tasks/task framework, across 197 new files. It is the second of three stacked PRs._safe_extract_tarnow usesfilter=\"data\", all threeEvaluationEngineexception paths (CancelledError,ExecutionError, infrastructure failure) are shielded, andBudgetLedger.reserve/refundcorrectly propagateCancelledErrorthrough best-effort rollback. Themax_proposalsdead-branch validation guard is also fixed.InferenceUsageStore._save()blocks the event loop on everyreserve()andcomplete()call \u2014 a synchronous disk write insideasync with self._lock. Under high concurrency this serializes all gateway coroutines behind two file writes per request. The analogousBudgetLedgerusesasyncio.to_threadfor the same pattern; aligning them would be the natural fix.Confidence Score: 5/5
Pure-add PR with no modifications to existing code; all issues flagged in the previous review have been addressed in this revision.
The only finding is a performance concern in the inference gateway synchronous disk I/O pattern, which does not affect correctness or security.
Files Needing Attention: vero/src/vero/harbor/inference.py — the synchronous _save() pattern inside the async lock is worth revisiting before this runs under high concurrency load.
Important Files Changed
Reviews (18): Last reviewed commit: "Replace EvaluationModel with a shared St..." | Re-trigger Greptile