feat(farm): best-of-N sampling + iterative retries + sampling params (first-time-go, Slice 1)#137
Merged
Conversation
Slice 1 of the 2026-06-26-farm deep-review. Opt-in worker-quality levers that spend the cheap worker-token axis; FARM_SAMPLES=1 is behaviorally unchanged. - F1 best-of-N (FARM_SAMPLES): N candidates drawn in parallel in isolated scratch worktrees, the first to pass the deterministic gate wins; a shared FARM_CONCURRENCY limiter caps TOTAL in-flight worker calls (never N x it). Token spend summed across samples + the accepted candidate's own tokens surfaced in farm-report.json. - F4 sampling params: temperature (auto-bumped off 0 when N>1 so samples diversify) and an optional max_tokens cap on the worker call. - F2 iterative retries: the worker is shown its own prior in-scope output (through the existing byte-cap + secret-redaction chokepoint), not just the gate-failure tail, so it refines rather than restarts. 167 tests green (incl. a real-worktree best-of-N smoke test + an N=1 regression pin); typecheck clean; farm.js rebuilt. F8 (worker sandbox) descoped to an ADR; F3 (auto-context) is Slice 2. Claude-Session: https://claude.ai/code/session_011nnQYBPd5Dt1rmtf9CHNfU
…review report Governance trail for the best-of-N sprint: the approved Slice-1 spec and plan, the append-only SMARTS decision log, and the docs/reports/2026-06-26-farm deep-review report that seeded the scope (F1/F2/F4 in; F8 -> ADR; F3 = Slice 2). Claude-Session: https://claude.ai/code/session_011nnQYBPd5Dt1rmtf9CHNfU
…line, input guards Independent review (security PASS, correctness SHIP; no BLOCK/HIGH) surfaced two MEDIUM + LOW items, all applied: - M1: a best-of-N sample that THROWS now resolves to a failure outcome (per-sample try/catch), so Promise.all never rejects and the scratch-worktree cleanup loop always runs — no worktree leak on the exception path. - M2: samples are cut from the TASK branch (a frozen integration-at-task-start snapshot), not the live farm/integration ref, so every sample shares the exact baseline the task worktree re-gates and merges against (AC-F1.2) — removes a false-escalation window when a sibling task merges mid-flight. - L1: non-numeric FARM_SAMPLES falls back to 1 instead of NaN (which would empty every sample batch and silently mass-escalate the run). - L2: an explicitly-set FARM_TEMPERATURE=0 is no longer auto-bumped to 0.7. - L3: prior-attempt context is shown only when the prior worker actually wrote files (no mislabeling inherited baseline as "your previous attempt"). - security LOW: worker-error and setup notes are wrapped in redactSecrets for parity with the gate/merge notes. 169 vitest green (+2: throwing-sample, non-numeric-samples); typecheck clean; farm.js rebuilt. Claude-Session: https://claude.ai/code/session_011nnQYBPd5Dt1rmtf9CHNfU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Slice 1 of the
docs/reports/2026-06-26-farm/deep review — opt-in levers to raise the--farmworker's first-time-go acceptance, spending only the cheap worker-token axis.--farmstays a Feature Forge preview (off by default);FARM_SAMPLES=1+ default knobs are behaviorally unchanged (pinned by a regression test).F1 — best-of-N against the gate (
FARM_SAMPLES, default 1)The gate is a deterministic pass/fail oracle and each task runs in an isolated worktree, so N candidates are drawn in parallel (each in its own scratch worktree cut from the task's frozen baseline) and the first to pass the gate wins; the winner's files are taken into the task worktree and merged, losers discarded. A shared limiter caps total in-flight worker calls at
FARM_CONCURRENCY— sampling shares the budget, never multiplies it.farm-report.jsonrecords both the summed sample-token spend and the accepted candidate's own tokens, so the N×-tokens trade-off is visible.F4 — sampling params (
FARM_TEMPERATURE,FARM_MAX_TOKENS)The chat body now carries
temperature(default 0; auto-bumped to 0.7 whenFARM_SAMPLES>1and temperature is unset, so samples diversify) and an optionalmax_tokenscap.F2 — iterative retries
On a retry (failed gate, or a sampling round with no green) the worker is shown its own prior in-scope output, not just the gate-failure tail, so it refines rather than restarts blind — through the same
FARM_ENRICH_MAX_BYTEScap + secret-redaction + secret-filename-denylist chokepoint as all injected context. Out-of-scope drift is never carried forward.Scope decisions (at the sprint gate)
ca-sandboxis a Docker/clone-into-container model that doesn't fit the farm's host-worktree architecture, and there's no agentic worker yet to protect; it'll be designed against the real consumer when Track 2 is scoped (ties to CONFIRM-05).Verification
farm.jsrebuilt and in sync; version bumped 2.5.2 → 2.6.0 (badge + CHANGELOG).Independent two-pass review (pre-PR)
assertSecureBaseUrlintact, no new egress..codearbiter/sprint-log.md).Generated autonomously via
/ca:sprint; full decision log in.codearbiter/sprint-log.md.https://claude.ai/code/session_011nnQYBPd5Dt1rmtf9CHNfU