Solidify RLE as a harness x model benchmark: scoring 1.2, swappable harness plugins, zero-Felix core, RimAPI MCP server - #52
Merged
Conversation
…coherence (#51) Both removed metrics were ~1.0 by construction (the resolver marks every detected conflict resolved; empty message traffic scored 1.0) and read Felix-specific counters, so they inflated every composite by ~20% and could not score a non-Felix harness fairly. - add scoring/coherence.py: contradiction detection over a tick's executed writes (draft/undraft, move+job, work_priority clash, overlapping zones, duplicate blueprint cell, competing research targets) - plan_coherence metric (weight 0.08); efficiency and plan_coherence return a neutral 0.5 for ticks with no writes so the unmanaged baseline earns no free process points - drop conflicts_*/messages_* from MetricContext; keep resolver + CentralPost counts on the CONFLICT event as diagnostics - redistribute weights (outcomes dominate); sync all 6 scenario YAMLs - SCORING_VERSION 1.1 -> 1.2 (pinned baseline sidecars now require recalibration) - README/CLAUDE/ADR-003 now match the code Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
…neHarness, entry-point registry RLEGameLoop now owns only the environment (pause/state/execute/score/export) and delegates decisions to a harness. The Felix stack (CentralPost wiring, MapAnalyst-first deliberation, per-agent timeouts, phase/score/error broadcasts, helix visualiser, generation-id accounting) moves verbatim into rle.harness.felix.FelixHarness; the unmanaged colony becomes BaselineHarness. The loop has no felix_agent_sdk import left. - rle.harness.protocol: BaseHarness, StepResult (plan + optional pre-applied execution for tool-using harnesses), HarnessContext, TickObserver, HarnessPlugin, Availability - rle.harness.registry: discovery via the rle.harnesses entry-point group, option validation against each plugin's pydantic schema, --harness-opt key=value parsing; baseline + felix registered in pyproject - rle.harness.compat: RLEGameLoop(agents=..., no_agent=...) keep working - FelixOptions (parallel, no_think, helix_preset, role_timeout_s, exclude_agent, provider_kwargs, visualize); provider/helix construction moves off RLEConfig into rle.harness.felix.provider_factory - RLEConfig gains harness / harness_options / tick_timeout_s; loop-level step timeout and HarnessStepError degrade to an empty scored tick - dashboard export gains harness + extras; TickResult records harness and step latency - tests: registry, compat, custom harness through the loop, pre-executed writes scored by plan_coherence, failure/timeout degradation Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
…ry; rle.testing for plugin authors Both run_scenario.py and run_benchmark.py now build their harness through rle.harness.create_harness and carry no Felix imports. --no-agent stays as a permanent alias for --harness baseline; --no-think/--sequential/--visualize fold into FelixOptions (warned and ignored for other harnesses). - rle.harness.cli: shared argparse glue + plugin table - run_benchmark.py: --harness is repeatable (harness matrix in one run); results and summaries record harness, harness_versions, mean step latency, and a harness_failed quarantine flag derived from RIMAPI null-ref / plant-def markers; quarantined runs are excluded from means - rle.orchestration.save_loader: load + settle helper shared by both CLIs (replaces the 2s sleep in the benchmark path) - rle.testing: MockRimAPI (moved out of run_benchmark.py) and run_harness_smoke(plugin) — the contract test external harness packages run in CI - collect_metadata: felix_sdk_version -> harness_versions (from BaseHarness.describe()) - docs/harness-plugins.md: authoring guide + repo boundary rule Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
With the extra uninstalled, RLE core imports, 406 tests pass, --harness list reports felix as unavailable with the fix hint, and --harness baseline runs end to end. Enforced by scripts/check_harness_boundary.py (felix_agent_sdk may only be imported under src/rle/harness/felix/; no third-party harness names in src/tests/scripts) and a new test-no-felix CI job. - move role agents, base_role and the claude-code provider under rle.harness.felix (git mv; rle.agents keeps only the neutral action vocabulary + json_repair); rle/__init__ no longer exports agents - ActionOutcome / ExecutionResult move to rle.agents.actions so the harness layer never imports rle.orchestration (broke a real cycle); action_executor re-exports them - rle.harness.brief: harness-neutral scenario brief (goals, state snapshot, MAP_SUMMARY, action catalog); base_role delegates its MAP_SUMMARY builder to it - RLEConfig loses helix_preset (FelixOptions has it); mcp optional extra - HF dataset card keyed by harness/model; run metadata harness_versions - tests: Felix-only modules are collect_ignore'd without the extra, requires_felix marker for individual cases; conftest fixtures probe lazily - CI: install .[dev,felix,mcp]; new test-no-felix and external-plugin-contract jobs; boundary check in lint - README/CLAUDE/CONTRIBUTING: harness x model framing, install extras, --harness flags Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
Tool-using harnesses act during their turn, so the environment cannot execute for them. The MCP server executes each write immediately through ActionExecutor (same normalisation and guards as Felix) and records it in a TickLedger; the harness drains the ledger into StepResult(execution=...) and the loop scores what actually reached RIMAPI. - rle.mcp.ledger / session: framework-free ledger + tool logic (writes outside a tick rejected; pawn-targeted writes without an id fail loudly instead of the executor's silent skip) - rle.mcp.server: MCPServer with get_brief / get_state / list_actions / list_reads / rimapi_read / end_turn plus one tool per WRITE_CATALOG entry - rle.mcp.host: in-process streamable-HTTP host (uvicorn, free port) so an external agent's MCP client and the loop share one ledger - rle-mcp console script: stdio server for manual play against a live game - optional extra mcp>=2.1; tests cover in-memory tool calls and a real HTTP round trip via mcp.client.Client Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
…agent harnesses rle.harness.cli_base.HeadlessCliHarness is the tool-agnostic base that harness packages wrapping a CLI coding agent subclass. It hosts the RLE MCP server in-process, builds the neutral brief and turn prompt, runs the turn protocol (prompt -> agent acts through tools -> end_turn or idle grace), applies turn_timeout_s, drains the ledger into StepResult(execution=...), and records latency, tokens and the deliberation log. Subclasses implement start_agent / send_turn / stop_agent only. rle.testing.scripted_agent.ScriptedMcpHarness plays a fixed tool script over a real MCP client connection; external plugins return it from smoke() so CI exercises the full round trip without their binary. Integration tests run it through RLEGameLoop against the mock RIMAPI and cover turn timeout / agent error degradation. Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
create_harness(smoke=True) no longer gates on plugin.available() — smoke harnesses exist so CI can exercise a plugin without its binary. A smoke that still needs a missing dependency (felix without the SDK) surfaces as HarnessUnavailableError with the plugin's own reason. src/rle/py.typed lets external harness packages type-check against RLE under mypy --strict. Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
…DR-004, docs - Leaderboard rows keyed by (harness, model); legacy history defaults to felix. Scenarios flagged harness_failed are excluded from means and counted; mean step latency and cost are Pareto columns; reads the cost_snapshot block the CLIs actually write. - update_baseline keys per harness x model and ignores quarantined scenarios. - ADR-004 records the swappable-harness decision, the repo boundary rule and the alternatives rejected. - CLAUDE.md / CONTRIBUTING.md architecture and package structure reflect the harness layer, MCP server, testing exports and the external harness repos. - CI contract job no longer relies on tee /dev/stderr. Co-authored-by: Jason <jkbennitt@users.noreply.github.com>
jkbennitt
marked this pull request as ready for review
September 4, 2026 14:34
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
RLE was built around one harness (the Felix 7-agent stack) and could only swap models. This makes the harness a first-class benchmark variable, swappable exactly like
--model, and fixes the scoring flaw that made every harness look the same.Tracks #51 (Phase C process metrics). Related: #6, #8, #46. Design record:
docs/adr/004-swappable-harnesses.md.What changed
Scoring 1.2 (#51)
coordination/communication_efficiencywere ~1.0 by construction (resolver marks every conflict resolved; empty message traffic scored 1.0) and read Felix-only counters. Removed.plan_coherence: 1 - contradictory executed writes / executed writes per tick (draft/undraft, move+job, work-priority clash, overlapping zones, duplicate blueprint cell, competing research targets). Computed from what reached RIMAPI, so any harness is scored the same way.efficiencyandplan_coherencereturn a neutral 0.5 on ticks with no writes; the unmanaged baseline no longer banks free process points.SCORING_VERSION1.1 -> 1.2; the pinned Crashlanded baseline sidecar needs recalibration against a live game.Harness layer
rle.harness:BaseHarness/StepResult/HarnessContext.RLEGameLoopowns only the environment and callsharness.step(...); legacyagents=/no_agent=still work viarle.harness.compat.FelixHarness(CentralPost, MapAnalyst-first, per-agent timeouts, phase/score/error broadcasts, helix visualiser, generation-id accounting moved verbatim out of the loop) andBaselineHarness.rle.harnesses): built-ins and third-party packages register identically.--harness NAME,--harness list,--harness-opt key=value(validated by each plugin's pydantic schema),--no-agentkept as alias.run_benchmark.py --harness a --harness bruns a matrix. Smoke variants run without the external tool.StepResult.executionlets tool-using harnesses report writes they already applied; the loop skips its executor and scores them.Zero-Felix core
felix-agent-sdkis now the optionalfelixextra. Role agents, base_role and the claude-code provider moved underrle.harness.felix/(git mv).rle.agentskeeps only the neutral action vocabulary (Action,ActionPlan,ActionOutcome,ExecutionResult).--harness listshows felix unavailable with the fix hint,--harness baselineruns end to end. Enforced byscripts/check_harness_boundary.py+ atest-no-felixCI job.py.typedshipped so plugins type-check undermypy --strict.rle.harness.brief: harness-neutral scenario brief (goals, state, MAP_SUMMARY, action catalog).RimAPI MCP server (
mcpextra) + coding-agent scaffoldrle.mcp: one tool perWRITE_CATALOGentry executing immediately throughActionExecutorinto a per-tickTickLedger, plusget_brief/get_state/rimapi_read/end_turn. In-process streamable-HTTP host so an external agent's MCP client and the loop share one ledger.rle-mcpstdio script for manual play.rle.harness.cli_base.HeadlessCliHarness: tool-agnostic base for CLI coding agents (turn protocol,turn_timeout_s, idle grace, ledger drain, cost/latency).rle.testing.scripted_agent.ScriptedMcpHarnessplays a fixed tool script over a real MCP client so plugin CI covers the full round trip without the binary.External harness repos (created in this PR's scope)
external-plugin-contractjob).opencode serve+ HTTP session API, MCP viaOPENCODE_CONFIG_CONTENT,permission: allow.grok -p --output-format json, session resumed per tick, project-scoped.grok/config.tomlMCP.All three install from GitHub and run
--smoke-testthrough the RLE CLI with zero core changes; the two coding-agent packages also drive their real code paths against fake servers/binaries in their own tests.Hygiene
rle.orchestration.save_loader) instead of the benchmark's 2 s sleep.harness,harness_options,harness_versions, per-tick step latency, and aharness_failedquarantine flag (RIMAPI null-ref / plant-def markers).s/stepand cost as Pareto columns;update_baselinekeys per harness x model.Repo boundary rule
Only RLE-authored harnesses (
baseline,felix) live here. Harnesses wrapping third-party tools ship as their ownAppSprout-dev/rle-harness-*packages.check_harness_boundary.pyfails CI on strayfelix_agent_sdkimports or third-party harness names insrc/,tests/,scripts/.Verification
pytest585 passed,ruff,mypy --strict, boundary check: green--harness list+ baseline smoke OKrun_benchmark.py --smoke-test --harness felix --harness baseline, and with all three GitHub-installed plugins (template,opencode,grok-build) in one matrix: OKFollow-ups for you
rle-harness-template(not possible via the API token here).scripts/calibrate_baseline.pyagainst a live game to refresh the scoring-1.2 baseline sidecar.--harness opencode/--harness grok-buildwith the binaries installed.