Greenfield ML auto-research flow + engine support

[cgpadwick]

Stacks on #1. Adds the engine features needed for real ML auto-research flows, then a self-contained MLE-Beast-style greenfield pipeline that runs end-to-end and beats its target via hill-climbing.

Review by commit (5)

1. engine: reset nested-loop counters on subflow entry — a retry_loop nested in a counting_loop now gets a fresh attempt budget each outer iteration (Subflow.prep clears its own counters). Required for the implement→verify retry inside the hill-climb.
2. engine: configurable workspace root, venv auto-activation, append_file — tools/commands operate in a --workspace dir (decoupled from where the flow lives, {{ workspace }}); every command auto-activates the workspace venv once it exists (no source in prompts); new append_file tool.
3. docs: greenfield ML design sketch — maps MLE-Beast's pipeline onto cwe primitives; locked-in decisions (user-provided metric direction, LLM-written train harness, base-stack setup, greenfield-only).
4. engine: expose {{ flow_dir }}; trim run-summary file list — flows can ship helper scripts a command invokes; run summary shows real outputs, not .git/.venv internals.
5. flows: greenfield ML auto-research flow (MNIST hill-climb) — flow.yaml + 8 skills + setup_env.py (ml-frameworks GPU stack / CPU fallback) + keep_or_revert.py (deterministic git keep/revert).

Verified end-to-end

Ran live on an RTX 4070 with deepseek/deepseek-v4-flash via OpenRouter (torch 2.3.1+cu121):
init → baseline 0.9713 → keep 0.9813, beating the 0.97 target via a real hill-climb iteration; git_init/baseline_commit (prose skills) and keep_or_revert.py (Python) all behaved.

Tests

Engine suite: 30 passing (unit + integration, offline). CI runs them on push/PR across Python 3.10–3.12. The ML flow is live-only (needs a provider + GPU), not part of the offline suite.

Notes

• Setup uses the CUDA-only cgpadwick/ml-frameworks stacks on GPU boxes; a lean CPU torch fallback keeps it portable.
• git_init/baseline_commit are LLM agents (per request, readable skills) — trade a little determinism for prose; keep_or_revert stayed deterministic Python.

🤖 Generated with Claude Code

[cgpadwick]

Code Review — Greenfield ML auto-research flow + engine support

Stacks on #1 (base initial-development). Two cleanly separable halves: small, well-tested engine additions (configurable workspace, venv auto-activation, append_file, nested-loop counter reset, {{ flow_dir }}), and a greenfield ML flow (8 skills + setup_env.py + keep_or_revert.py). The engine half is excellent; the flow half has one integrity issue worth addressing before relying on its results.

Engine changes — strong, and they close prior gaps

• Nested-loop reset (Subflow.prep + reset pairs) is the right fix and is well-targeted: it also clears _feedback[action_id] on retry-loop entry, which resolves the stale-feedback nit I raised on #1. The test_nested_retry_loop_resets_each_outer_iteration test (6 = 2×3, not 4) pins exactly the bug. Good.
• venv_env existence-gate is a genuinely elegant bootstrap solution: the command that creates the venv runs with system Python, everything after auto-activates it — no source in prompts, fully deterministic. Tests cover present/absent/PATH-ordering.
• Workspace decoupling is clean; resolution order (arg → YAML → flow dir) is documented and back-compat preserved.

Issues & risks

1. (Most important) The score is self-reported by the LLM, not read from the source-of-truth file — it's gameable and diverges from your own design. evaluate/baseline_eval capture best_score/candidate_score via set: regex over the agent's free-text reply ("Test accuracy:\\s*..."). The evaluate skill tells the agent to run evaluate.py, but nothing forces it to — the agent can emit Test accuracy: 0.99 without running anything, or edit_file evaluate.py to print a fixed number (it has edit_file in its tool list and is instructed to "fix it and rerun" until a score appears). Since loop termination (exit_when: best_score >= target) keys off this number, the model is structurally incentivized to produce a passing line. Your own design doc (§1, §6) says "Score = source of truth file: eval_results.json" — but the implementation reads prose instead. Recommend: capture the metric from a deterministic command step, e.g. python -c "import json;print('SCORE='+str(json.load(open('eval_results.json'))['value']))", and set: off that command's stdout. That restores the file-as-truth contract and removes the gaming surface. (flows/greenfield_ml/flow.yaml evaluate/hc_eval, evaluate/skill.md)

2. Score-scale mismatch silently passes the target. [0-9]*\\.?[0-9]+ happily captures Test accuracy: 98 → best_score=98, and 98 >= 0.97 exits the hill-climb on iteration 0 as "target met." Reading from eval_results.json (a known float in [0,1]) per #1 also fixes this; otherwise consider validating the captured value's range. (flows/greenfield_ml/flow.yaml)

3. Supply-chain / arbitrary execution in setup_env.py. It clones cgpadwick/ml-frameworks (--depth 1, then git pull --ff-only on re-run) and runs poetry install, which executes third-party build code into the workspace venv. It's your own repo so this is acceptable, but worth a one-line comment/README note that setup runs remote code, and pinning the clone to a tag/SHA rather than tracking the default branch would make runs reproducible. (flows/greenfield_ml/setup_env.py:install_gpu_stack)

Minor / nits

• keep_or_revert.py has no test, yet it's the deterministic core of the hill-climb (improve→commit→reset-failures vs revert→increment). A tiny unit test over the four branches (improved/not, lower_is_better true/false, failure-counter math, research-log preservation across git clean) would be cheap insurance — especially the research-log save/restore, which is subtle. (flows/greenfield_ml/keep_or_revert.py)
• improved = cand > best treats ties as regressions (revert + failure++). Fine, but worth a comment since a re-run yielding the identical score will count as a plateau step.
• venv_env is Unix-only (bin, not Scripts). Fine for Linux/WSL; note it if Windows is ever a target. (cwe/tools.py:venv_env)
• Workspace path not resolved (Path(workspace or ...).expanduser()) while flow_dir is .resolve()d into the seed — a relative --workspace leaks a relative {{ workspace }} into prompts/commands. setup_env.py re-resolves it, so it works today, but resolving once in build_flow would be more consistent. (cwe/hydrate.py)
• Nested-loop _iter[name] reset means the run-summary under-reports total iterations for nested loops (only the last entry's count survives). Cosmetic. (cwe/primitives.py + cli._print_summary)
• git_init/baseline_commit moving from deterministic git to LLM agents is acknowledged in the PR; reasonable given the prose-readability goal, just trades a little determinism on a step that can't really go wrong.

Test coverage

Engine additions are well covered (workspace x3, venv x3, nested reset, append_file) — 30 passing offline. Gaps: the two shipped Python helpers (keep_or_revert.py, setup_env.py) have no unit tests despite being deterministic and testable (see nit above), and the ML flow is live-only (acknowledged, reasonable).

Verdict: Engine half is clean, tested, and closes real gaps — merge-ready. For the flow, I'd fix #1 (read the metric from eval_results.json via a command, not the agent's reply) before treating any "beat the target" result as trustworthy; the rest are minor. Nice work overall.

🤖 Generated with Claude Code

[cgpadwick]

this is really weird, why aren't we using argparse here??!?!

[cgpadwick]

Agreed — rewritten with argparse in 3ecfeb8: a real run subcommand, typed options, mutually-exclusive -v/-q, and a free --help. The hand-rolled parser is gone.

[cgpadwick]

this looks like old c code lol

[cgpadwick]

Ha, fair :) that index-walking while i < len(args) loop is exactly the C-ism — argparse handles all of it now (3ecfeb8).

[cgpadwick]

isnt this the same as in the .gitignore? or does this become the gitingore file?

[cgpadwick]

Good question — it is neither. It is a display-only filter for the run summary's "files written:" line, so it does not list hundreds of .git/.venv/data internals. It is never written to disk and is not a .gitignore; the overlap with common ignore patterns is just because those dirs are noise either way. Renamed to _SUMMARY_SKIP_DIRS with a clarifying comment in 3ecfeb8.

[cgpadwick]

Thanks — worked through the review. Status:

Inline comments (cli.py) — fixed in 3ecfeb8:

• Replaced the hand-rolled index parser with argparse (real run subcommand, typed options, -v/-q, free --help).
• Renamed _SNAPSHOT_SKIP → _SUMMARY_SKIP_DIRS with a comment clarifying it's a display-only filter for the run-summary's "files written" line (not a .gitignore, never written to disk).
• Added tests/test_cli.py.

#1 / #2 (score read from prose, gameable + scale-blind) — fixed in b227ee2:

• evaluate/hc_eval now run evaluate.py then read_score.py, which reads eval_results.json['value'] (the documented source of truth), validates it's a finite fraction in [0,1], and prints SCORE=<value> only then. A missing/malformed/out-of-range value yields no capture, leaving the score untouched instead of spuriously meeting the target (Test accuracy: 98 no longer "passes"). Added tests/test_read_score.py.

Already addressed earlier on the branch:

• Retry LLM calls with exponential backoff #3 supply-chain — setup_env.py has --ref pinning + a "fetching/executing remote code" warning (47a0415).
• Workspace .resolve() consistency and the venv_env POSIX-only note (47a0415).
• keep_or_revert.py unit test (47a0415); nested-loop reset + stale-_feedback clearing (760c9b9).

Suite is at 42 passing offline. Remaining items are the acknowledged cosmetic nits (nested-loop run-summary under-count; git_init/baseline_commit as agents).