Skip to content

v3.17.0 β€” MetaHarness learn + GEPA: harness policies become measured, evolved artifacts

Choose a tag to compare

@ruvnet ruvnet released this 03 Jul 18:24

v3.17.0 β€” MetaHarness learn + GEPA: harness policies become measured, evolved artifacts

This release closes the harness-policy lifecycle loop: learn a policy from a scored task corpus β†’ audit what it compiles to β†’ evolve it against fixed benchmarks β†’ analyze why runs fail. Two new MCP tools (13 β†’ 15), two new CLI subcommands (11 β†’ 13), two new skills, and version-pin upgrades across the @metaharness/* family.

New: metaharness learn β€” learned policies instead of hand-tuned prompts

Wraps upstream metaharness@0.3.0's ADR-235 GEPA learning harness. Instead of a human editing a system prompt and eyeballing the result, learn mutates the harness genome, scores candidates against held-out SWE-bench-style slices, and only promotes measured winners.

# $0 dry-run β€” resolves the slice and prices the run, no model calls
npx ruflo metaharness learn --host claude-code --model haiku --slice slices/lite.json

# real run β€” explicit spend opt-in at every layer
npx ruflo metaharness learn --repo ~/src/metaharness --host claude-code \
  --model haiku --slice slices/lite.json --run --timeout-ms 1800000
  • $0 by default β€” spending requires --run (CLI) / run: true (MCP tool metaharness_learn)
  • Needs a metaharness repo checkout (--repo or $METAHARNESS_REPO); without one you get a structured {status: "checkout-required"} payload with clone instructions β€” a precondition report, not an error

New: metaharness gepa β€” genome inspection and auditing

Surfaces the @metaharness/darwin@0.8.0 GEPA library entry (@metaharness/darwin/gepa) β€” the first plugin surface with no CLI binary behind it.

npx ruflo metaharness gepa --op genome      # load + validate the shipped cand-6 genome
npx ruflo metaharness gepa --op render      # genome β†’ the system prompt it compiles to
npx ruflo metaharness gepa --op validate --path my-genome.json --alert-on-invalid   # CI gate
npx ruflo metaharness gepa --op analyze --transcript run.json   # failure-class breakdown

Why it matters: evolved policies ship as genome JSON, but the behavior lives in the system prompt the genome compiles to. --op render is the "what does this policy actually say?" view you want before adopting one. The shipped cand-6 genome is the first holdout-confirmed promotion (holdout gold 2/12 β†’ 3/12, zero regressions, empty-patch rate 0.583 β†’ 0.333).

Deliberately not surfaced: gepaOptimize β€” it takes an in-process bring-your-own-evaluator callback that can't cross a subprocess boundary. Import @metaharness/darwin/gepa directly, or use metaharness_evolve for sandbox-scored evolution.

Version pins refreshed

Package Before After
@metaharness/darwin ~0.3.1 ~0.8.0 (GEPA engine, new evolve selection modes: pareto, clade, niche-steering, quality-diversity, behavioral-diversity, plus --crossover / --epistasis / --curriculum / --mutator ruvllm)
metaharness ~0.2.6 ~0.2.8 (learn arrives via @latest β†’ 0.3.0)
@metaharness/redblue ~0.1.1 ~0.1.4
@metaharness/kernel ~0.1.0 ~0.1.2

Fixes

  • redblue stale-cache trap: the install cache dir is now versioned by the pin (~/.ruflo/redblue-cache-0.1.4), so pin bumps invalidate stale installs β€” previously a machine that cached 0.1.1 would serve it forever
  • gepa stale-install fallback: a pre-0.8.0 darwin in an ancestor node_modules throws ERR_PACKAGE_PATH_NOT_EXPORTED (no ./gepa subpath); gepa.mjs treats that as recoverable and falls back to its versioned cache

Validation

  • tsc clean build
  • MCP runtime contract: 171/171 assertions across all 15 tools
  • Plugin smoke suite: 120/120 steps
  • darwin 0.8.0 evolve / bench create|verify / security bench proven through the wrappers; redblue 0.1.4 attack preview proven via the new cache path

All ADR-150 constraints hold: @metaharness/* stays in optionalDependencies, every path degrades gracefully, ruflo remains fully operational with MetaHarness absent.


Deep-dive: full capabilities walkthrough (usage + rationale) β†’ https://gist.github.com/ruvnet/cdc1bc2259cde127a418f861c3907e1e
PR: #2540