docs(research): rvagent Hermes-class harness architecture (metaharness + ruflo integration) - #752
Merged
Merged
Conversation
…rness + ruflo integration Research synthesis from four parallel investigations (Hermes harness web research; code audits of crates/rvAgent, ruvnet/metaharness, ruvnet/ruflo): - Findings: Hermes architecture/benchmarks, rvAgent's 4 blocking loop defects vs its production-grade A2A/MCP/security layers, metaharness Darwin/flywheel apparatus, ruflo coordination-plane seams - Target architecture: event-streaming loop, cache-first prompt tiers, compaction with lineage, 4-layer memory, skill synthesis learning loop, explicit rvagent<->ruflo and rvagent<->metaharness integration contracts - Phased roadmap (P0 foundation repair -> P4 evolution/SOTA) with falsifiable exit gates and six proposed ADRs Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…ness (P0.2, P0.4) Roadmap: docs/research/rvagent-hermes-harness/03-roadmap.md Phase 0. - rvagent-core: add ToolDefinition; ChatModel::complete/stream and StreamingChatModel::stream_chunks now take the active tool set; ToolExecutor::definitions() advertises schemas to the loop - rvagent-core graph: tool errors feed back to the model as tool results instead of aborting the loop; parallel branch now uses parallel_execute_limited (JoinSet + semaphore, bounded by new GraphConfig::max_parallel_tools) instead of sequential awaits; per-turn usage aggregated from provider metadata and logged - rvagent-backends anthropic: tools field on ApiRequest; usage and stop_reason attached to response message metadata - rvagent-backends gemini: full function-calling support (functionDeclarations, functionCall parts -> ToolCalls with synthesized IDs, tool results as functionResponse parts with name recovery); usage metadata - rvagent-cli: CliToolExecutor advertises builtin tool schemas - tests: tool serialization, usage metadata, error feedback, parallel failure isolation, Gemini function-call roundtrips Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…ine (P0.1, P0.3) Roadmap: docs/research/rvagent-hermes-harness/03-roadmap.md Phase 0. P0.1 — one canonical type system: - rvagent-core is now the only source of AgentState/Message/ToolCall/ TodoItem/TodoStatus/RunnableConfig; middleware's duplicate definitions deleted (new types.rs re-exports core + keeps middleware envelopes) - rvagent-subagents' HashMap-based AgentState alias replaced with the typed core AgentState (ADR-103 A1), preserving isolation semantics - ToolMessage gains optional tool_name; graph pushes tool results with the tool name so sanitizer middleware can label output - middleware Tool trait aligned with core ToolDefinition (input_schema(), definition()) P0.3 — middleware actually runs: - Middleware::before_agent/wrap_model_call now async; sync/async duplicate hooks (abefore_agent/awrap_model_call/AsyncModelHandler) deleted; MiddlewarePipeline drives the async chain so real HTTP model calls can run inside it; RetryMiddleware uses tokio sleep - new PipelineModel<M: ChatModel> adapter runs modify_request + the wrap_model_call chain around any inner model - rvagent-cli invoke_agent and rvagent-acp prompt now build the default pipeline from configured middleware names and run all model calls through it (previously zero middleware executed on these paths) Net -242 lines. Verified: full rvAgent workspace test sweep green. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
- README: 10-crate map (adds rvagent-mcp, rvagent-a2a); parallel-tools claims now describe the actual bounded-concurrency JoinSet behavior; removed unverified 5-10x and HNSW O(log n) semantic-search claims (hnsw middleware is an experimental hash-embedding placeholder) - architecture.md: adds mcp/a2a to the dependency graph and an explicit current-gaps note (no SSE streaming, stubbed subagent spawning, placeholder hnsw embeddings) - middleware doc comments: drop inherited 150x-12,500x claim Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
Findings from a security review of the Phase 0 diff: - MEDIUM: a panicking tool aborted the entire agent process. Tool args are model-controlled and indirectly attacker-influenced via prompt injection, so any panic on crafted input was a DoS. Each parallel tool call now runs in a nested spawn; JoinError becomes a normal tool-error result on the model's recovery path. Regression test added. - MEDIUM: RetryMiddleware treated empty content as a transient error. Now that tool schemas are actually sent, tool-call-only turns legitimately have empty text and would be retried up to max_retries (4x token cost, valid responses discarded). Empty content now only counts as an error when tool_calls is also empty. - MEDIUM: the Gemini API key was passed in the URL query string, where it lands in reqwest error Displays (which are logged and now also surface into the middleware chain) and proxy logs. Moved to the x-goog-api-key header. - LOW: synthesized Gemini call IDs (index+name) collided across turns, letting one turn's tool result satisfy another turn's call in id-keyed middleware. IDs now use a process-wide atomic counter; functionResponse name resolution prefers the recorded tool_name over the id fallback. Verified safe, no change needed: tool errors go through the same tool_result_sanitizer as success output on both CLI and ACP paths; HITL-denied tool calls stay denied through PipelineModel reattachment; COW state gives executors immutable snapshots (no torn reads); max_parallel_tools bounds concurrency; advertised tool schemas are static literals with no config or secret material. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…confinement
Adds the Phase 0 exit-gate test: an end-to-end verification that wires the
real builtin tool registry, the real filesystem backend, and the real
AgentGraph loop against a scripted model. Only the provider network call is
faked, so the gate fails if the loop, the schemas, or the tools regress.
Writing the gate surfaced two problems.
1. The filesystem backend applied no path confinement. Tool-supplied paths
were passed straight to std::fs, so a model-issued read_file or write_file
with an absolute path (or ../ traversal) escaped the working directory
entirely. Paths from the model are untrusted input and were treated as
trusted. LocalFsBackend now resolves every path against a fixed root:
lexical .. normalization first, then canonicalization of the deepest
existing ancestor so a symlink inside the root cannot bridge out, then a
containment check. Escaping reads and writes are refused before touching
the filesystem.
2. Two of the gate's own assertions passed vacuously — a wrong parameter name
produced "file_path is required", which satisfied a loose contains("error")
check while never exercising the boundary under test. The confinement
assertions now require the specific refusal message.
The backend moves from the CLI binary into rvagent-tools so the gate exercises
the shipped code path rather than a lookalike copy; the CLI now consumes it
(-336 lines of duplication).
Covers: schemas advertised on every turn including after tool results, real
side effects on disk, tool errors feeding back as results without aborting the
loop, unknown tools reported not fatal, parallel execution preserving call
order and agreeing with sequential, usage metadata aggregation, and
confinement holding through the full loop.
Tests: 94 unit + 10 e2e in rvagent-tools; rvagent-cli/core/middleware/acp green.
Co-Authored-By: claude-flow <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
Five-area research sweep (benchmarks, harness techniques, competing architectures, long-horizon context, self-improving harnesses) with the roadmap corrections each finding implies. Findings that contradict the current roadmap: - Compaction: simple observation masking matches or beats LLM summarization at ~half the cost, and summarization inflates trajectories 13-15% by destroying stopping signals. Phase 1.3 and Phase 2.5 both bet on summarization, as does the shipped default middleware pipeline. - Subagents: Phase 1.6's CoW fork/merge + CRDT join is parallel-writer architecture, the one multi-agent pattern with strong negative evidence for coding. The patterns with production evidence are a fresh-context reviewer and a read-only context-gatherer. - Learning loop: the 2026 literature turned against trajectory-learning memory. Controlled baselines show agent self-memory underperforming plain retrieval, and a documented inverted-U where utility drops below no-memory. "SONA on the default path" needs a gate and a permanent control arm. - Positioning: three major 2026 harnesses are already Rust (Codex CLI, Grok Build at ~844k LOC open-sourced 2026-07-15, Goose). "Fast Rust harness" is not a differentiator. Two positions remain open: a stable embeddable library API with open governance, and deterministic replay. - Phase 4's SWE-bench-Verified gate targets a saturated benchmark whose noise band exceeds the effects we would claim. Missing from the roadmap entirely: programmatic tool calling (the only context strategy positive across all models tested), and most of the reliability floor (observation-window management, loop detection, environment bootstrap, persisted thinking) where the largest measured deltas live. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
… progress A stuck agent repeats the same call forever, and max_iterations does not fix that — it only makes it more expensive. The loop now fingerprints each tool call by (name, args) and refuses it once it has repeated consecutively past a threshold (default 3), substituting an actionable message that tells the model to change approach rather than an opaque error. Evidence: the harness technique literature ranks loop/stuck detection among the highest value-per-effort items, identifying repeated near-identical actions as the clearest failure signal in agent trajectories, and notes explicitly that raising iteration caps does not help. Counting is consecutive rather than windowed, deliberately. An agent that re-runs the same check between edits is doing legitimate work; a windowed counter would refuse it. Only an unbroken run of identical calls trips the detector. Alternating cycles are not caught — max_iterations remains the backstop, and the limitation is documented on the type. Refused calls still emit exactly one tool result each, in the model's original call order, so the provider's tool_use/tool_result pairing stays in sync. Tests cover: refusal after threshold with execution actually stopping, differing args not treated as a loop, interleaved re-runs not refused (the false-positive case that drove the design), disabling via threshold 0, and fingerprint stability across call ids and JSON key order. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
Five ADRs converting the 2026 SOTA sweep into decisions. Three of them reverse or constrain standing bets in the roadmap. ADR-273 Harness Reliability Floor. Sequences work by measured reliability impact. The cleanest ablation available moves +54.3 points, almost entirely from patch-apply failures dropping 69.1% -> <1.5%: the dominant wins are mechanical, not reasoning. Defines seven floor mechanisms and an explicit not-doing list (few-shot, ungrounded reflection, embedding code index, context beyond ~128k, learned components). ADR-274 Context Management: masking over summarization. Reverses Phase 1.3, Phase 2.5/ADR-252, and the shipped default pipeline. Observation masking matches or beats LLM summarization at ~half cost, and summarization inflates trajectories 13-15% by destroying stopping signals. Adds addressable recall, programmatic tool calling (the only context strategy positive on every model tested), verbatim invariant re-injection against documented governance decay, and per-model-tier capability gating. ADR-275 Subagent Topology: single writer with auxiliary intelligence. Reverses Phase 1.6's CoW fork/merge + CRDT join, which is parallel-writer architecture -- the one multi-agent pattern with strong negative evidence for coding. Adopts fresh-context reviewer (deliberately no shared context) and read-only context-gatherer on a cheap model tier. Keeps JoinSet/semaphore concurrency; drops the mergeable state type entirely. ADR-276 Learning Loop: gating, trust tiers, measurement. Constrains "SONA on the default path". Ships feature-gated off with the measurement apparatus as a precondition: paired lift, previously-solved regression rate, permanent memory-off control arm, plain-retrieval baseline, confound controls. Immutable episodic storage, delta-only consolidation, trust tiers with structural enforcement that untrusted-derived memories cannot influence permission decisions, and sequential-test promotion instead of greedy accept. ADR-277 Positioning, Protocols and Claims. Invalidates the premise that a Rust harness is itself differentiating -- Codex CLI, Grok Build (~844k LOC, open-sourced 2026-07-15) and Goose are already Rust. Repositions on the two open gaps: a stable embeddable library API with open governance, and deterministic replay reported as action-match rate rather than reproducibility, since hosted inference is not reproducible even at temperature 0. Mandates MCP 2026-07-28 migration, makes ACP first-class, and withdraws the SWE-bench-Verified exit gate as a claim target. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
Implements the default context strategy from ADR-274. Old tool observations are replaced with compact placeholders before history is sent to the model; reasoning steps and actions pass through verbatim. Masking is a projection, not a mutation: AgentState::messages stays the complete append-only log, and each placeholder carries the tool_call_id as a recall handle, so elided content remains addressable. Placeholders name the tool and the elided byte count so the model can tell what it is missing. Also caps individual tool results at write time (ADR-273 3.3) with an explicit truncation marker, in both the parallel and sequential execution paths. An uncapped tool result can consume the context window in a single call. Writing the end-to-end test for the cap surfaced two defects in read_file's line formatter: - Lines over the length limit were truncated silently, leaving the model believing it had seen the whole line. Now marked with the omitted byte count. - The truncation sliced at a raw byte offset, which panics when a multi-byte character straddles it. A UTF-8 file with a long non-ASCII line would have crashed the tool. Now walks back to a character boundary. The existing formatter test asserted the silent-truncation behaviour, so it was updated rather than left encoding the bug; added coverage for the char-boundary cases. Tests: 12 e2e (2 new: masking reaches the model with the full log preserved, oversized output capped), plus unit coverage for masking, truncation, and multi-byte safety. 38 suites green across the five rvagent crates. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…memory -> policy Studied ruvnet/metaharness (@metaharness/flywheel@0.1.7, ADR-226/228/236) for reusable self-learning. Three consequences. 1. Do not build a promotion apparatus. The flywheel already implements what ADR-276 3.4 specified, more rigorously: a frozen conjunctive gate with a SHA-256 fingerprint proving it did not move, a holdout PLUS a frozen anchor never optimized against, Ed25519 receipts, independent replay verification, and a lineage DAG that re-bases on the promoted winner. It is deliberately host-agnostic -- Policy = Record<string,string> with Proposer/Evaluator as the only seams -- so there is no adapter impedance. Offline, so no Rust port. 2. Shift self-learning investment from memory to policy. These are different objects with opposite evidence: policy text (GEPA-style) is the best-evidenced optimizer in the 2026 sweep, while episodic memory accumulation shows an inverted-U and confound-sized gains. RuVector's weight sits on the memory side. ADR-276's gating is not repealed; new effort moves to policy evolution. 3. Adopt noopRate as a score axis. The default gate requires it to strictly improve -- a policy earns promotion by making the executor commit more, not just score higher. Two internal nulls are now binding. ADR-226: a read-only frontier advisor produced zero marginal gold-scored resolves at 5.4x cost while genuinely firing (33 advisories, 3 vetoes) -- independently corroborating the +0.4pp/5.8x figure already cited from the public literature. ADR-236: the flywheel mechanism was proven end-to-end on real SWE-bench and still produced no compounding lift because the base solver was too weak, confirming that a promotion engine cannot rescue an unreliable loop and that ADR-273's reliability-floor-first ordering is correct. ADR-275 3.1 is amended accordingly: the fresh-context reviewer is downgraded from adopted to gated. It was written on Cognition's production data without considering ADR-226. ADR-226 gave its advisor the full transcript where this reviewer sees only the diff, so it does not refute the design -- but it is the null the reviewer must beat before reaching the default path. 3.2 is unaffected and strengthened. Also identifies an upstream contribution: the flywheel gate is single-shot, so many generations against one holdout is uncontrolled multiple testing -- the regime where PACE measured 30-42% false commits. Frozen conjunctive gate AND anytime-valid sequential test is strictly stronger than either. Closes a gap where ADR-271 did not reference the flywheel at all. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…, ADR-274) Environment bootstrap (ADR-273 3.5). A cheap factual workspace snapshot is injected into the system prompt before the loop starts, so the agent does not spend its first turns discovering cwd, project kind, test command, and branch. Filesystem-only -- no process spawning -- so it is safe to run unconditionally. Two deliberate restraints. build_ok is None unless a caller checks it, and an unchecked build renders nothing at all rather than reading as passing; a guess here would be worse than silence because the agent would trust it. And a truncated directory listing announces how many entries it dropped, since a partial list that looks complete invites the agent to conclude a file is absent. Compaction invariants (ADR-274 2.2). Safety constraints and task statements erode through successive compaction cycles with no failure signal -- a documented mechanism, not a jailbreak. InvariantSet re-emits a small set of rules byte-identical after each compaction; they are never inputs to a summarizer and never masked. Re-inserting an id replaces rather than accumulates, so an updated objective cannot leave a stale copy in force. The prompt composition lives in core as EnvironmentSnapshot::augment_prompt rather than at the CLI call site: rvagent-cli is a binary crate, so anything assembled there is unreachable from a test. This keeps the seam covered. Tests: 160 core unit tests (was 143), including unchecked-build silence, announced truncation, byte-stable invariant round-trips across repeated renders, and no-op augmentation on an empty workspace. 38 suites green, clippy clean. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
Belongs with 9d11dbc, which added tempfile as a dev-dependency of rvagent-core for the environment-bootstrap tests. The lockfile update was missed because that commit staged only crates/rvAgent. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…274) Verify-after-write (ADR-273 3.1). LocalFsBackend now reads back every write and edit, comparing bytes against what was requested. std::fs::write returning Ok means the syscalls succeeded, not that the bytes are on disk and readable -- a full filesystem, a quota, a racing writer, or an unusual mount can all produce a successful-looking write whose content differs. Reporting success there is the worst outcome, because the agent proceeds believing the edit landed and every later step rests on a false premise. That is the failure class that dominates harness ablations. Costs one read per write, negligible against a model round trip. Both failure messages tell the model to re-read the file rather than just reporting that something went wrong. Addressable recall (ADR-274 3.2). Masking is now non-destructive in practice, not just in principle: a `recall` tool dereferences an elided observation by the recall id carried in its placeholder. It is served by the loop from the full message log rather than by a ToolExecutor -- executors do not have the log, and reserving the name in the loop means a workspace tool cannot shadow it. Advertised exactly when masking is active, since a masked observation the model cannot dereference is worse than no masking at all. Tests: 14 e2e (2 new -- recall round-trips content that was elided from the model's view, and an unknown recall id is actionable rather than fatal), plus backend coverage for verified write/edit, content mismatch, unreadable target, and empty/multibyte content. 38 suites green, clippy clean. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
The shipped default pipeline used LLM summarization as its compaction strategy, which ADR-274 decided against. Until now every run contradicted an accepted decision. Summarization is now gated behind PipelineConfig::enable_summarization, off by default, alongside the existing SONA/HNSW opt-ins. It is removed from the CLI's DEFAULT_MIDDLEWARE (11 -> 10). Observation masking in the agent loop is the default; measured comparisons put simple masking at or above LLM summarization on solve rate at roughly half the cost, and show summarization inflating trajectories 13-15% by destroying the stopping signals an agent uses to notice it has finished. The middleware is kept and still constructible by name, so the fallback is real rather than nominal. Its trigger drops from 0.85 to 0.75 of the token budget when enabled: context degrades well before the nominal limit, and compacting at 85% leaves too little headroom to be selective rather than desperate. Reading the implementation while demoting it confirms the direction. Its summarize() keeps only truncated Human messages and discards all AI reasoning and tool results outright -- considerably lossier than a summarizer that at least attempts to preserve decisions and unresolved issues. Two tests now assert the decision rather than the old shape: summarization is absent from DEFAULT_MIDDLEWARE, and enabling it adds exactly one middleware while remaining resolvable by name. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…lizers db411ee added a field to PipelineConfig but only ran `cargo test`, which does not build bench targets. `cargo clippy --all-targets` caught it; the commit went out anyway because the command chained the commit after an echo rather than gating on the lint result. Verified with `cargo clippy --workspace --all-targets`. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
ADR-277 §5.1 withdrew the SWE-bench-Verified gate without a replacement. This supplies one, and the research that produced it killed the obvious candidate: SWE-bench Pro was retracted by OpenAI on 2026-07-08 after an audit flagged 27.4% of its 731 public tasks broken automatically and 34.1% by five independent human reviewers. Two benchmark retractions in six months is the context every claim now lands in. Also ruled out: SWE-Lancer (archived), Aider polyglot (frozen since Nov 2025), LiveCodeBench (model benchmark, not harness-sensitive), OSWorld (self-reported, meeting-gated verification), and bare GAIA (30-50 point spread on identical tasks from scaffolding alone). Four conjunctive gates: 1. Terminal-Bench 2.1 at >=78.0% on a mid-tier model, 5 trials, CI half-width <=1.5pp, team-verified. Deliberately mid-board: 78% on a mid-tier model is a stronger result than 84% on a frontier model, and claiming the top would be overreach given CI widths. 2. The actual harness claim -- >=+4.0pp over Terminus 2 on an identical model, CI-disjoint, across >=3 models and >=2 vendors, with variance decomposition separating harness- from model-induced effects. Absolute pass rate is a joint model x harness measurement; only the fixed-model delta is ours. Requires publishing the falsification point where the delta vanishes. 3. Cost-normalized Pareto. Terminal-Bench publishes no cost column and HAL has paused submissions, so no operating cost-normalized agentic-coding board exists -- uncontested ground, and the natural claim for a Rust harness. 4. SWE-rebench on the rolling window, where memorization is structurally impossible, to prove the result is not terminal-specific or contaminated. Adds required caveat language for every published number and a data-hygiene note: SEO aggregators are publishing figures that do not appear on primary boards, using real model names. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
… (ADR-273) "Error: old_string not found" is the highest-frequency tool failure in an editing agent and is nearly useless on its own: the model already believed the string was there, so restating that it isn't gives it nothing to change. It then retries a near-identical call, which is precisely the input condition for loop detection. The edit path now works out WHY the match failed and says so, checking causes in order of real-world frequency: CRLF-vs-LF line endings, whitespace (indentation, trailing spaces, tabs-vs-spaces), case, and finally the nearest line in the file echoed back verbatim so the model can copy the real text. When nothing is close it says that plainly and tells the agent to read the file rather than assume its contents. No new tool. edit_file already implements str_replace semantics, and ADR-273 3.4 makes the 8-15 tool budget something to defend rather than spend -- the published reproductions show the ergonomics are what move the number, not the tool's existence. Writing the end-to-end test corrected a wrong assumption: omitting leading indentation does NOT fail, because matching is substring-based. The failure that actually happens is supplying the WRONG indentation. Both the e2e test and a misleading unit-test comment were fixed rather than left describing a scenario that cannot occur. Tests: 111 unit + 15 e2e in rvagent-tools, including CRLF, tabs-vs-spaces, case, nearest-line, multibyte safety, and a guard that a trivially short needle does not produce a confident wrong guess. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
The Rust half of the flywheel seam: the thing being evolved (PolicyGenome) and the thing being measured (Score). rvAgent does not implement a promotion engine -- @metaharness/flywheel already provides the frozen fingerprinted gate, holdout plus never-optimized-against anchor, Ed25519 receipts, replay verification, and lineage DAG. PolicyGenome is shaped as the flywheel's Policy = Record<string, string> so there is no adapter impedance, and is backed by a BTreeMap so serialization is deterministic -- iteration order feeds gate fingerprints and replay, and a nondeterministic order would make identical genomes hash differently. It maps onto real config: max_iterations, parallel_tools, max_parallel_tools, loop_repeat_threshold, keep_last_observations, max_tool_result_bytes, plus two text levers for the prompt layer. Unknown levers are rejected rather than silently skipped. A mutation to a lever the harness does not apply produces a run identical to baseline, which the optimizer would score as "no effect" and burn generations on. Score projects runs onto the gate's four axes, including noopRate -- the non-obvious clause the default gate requires to strictly improve. A run that reports success while committing nothing counts as a no-op, so a policy cannot earn promotion by making the agent talk rather than act. Fixes a real bug at the JS seam found while testing: cost-per-win with zero wins was f64::INFINITY, which serde serializes as JSON null, and the gate's `candidate.costPerWin > baseline.costPerWin` evaluates `null > n` as false in JavaScript -- so a policy that won nothing would silently PASS the cost clause. Verified by running the gate's own logic against both encodings: null does not fire the clause, the finite sentinel does. Now uses f64::MAX as an explicit COST_PER_WIN_NO_WINS constant, with a test asserting no axis ever serializes non-finite. Tests: 17 policy tests including deterministic serialization, unknown-lever rejection, a guard that KNOWN_LEVERS cannot drift out of sync with apply_to, and the non-finite serialization guard. 174 core tests green. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…DR-275) Implements the subagent boundary as a tool that spawns an isolated context and returns a String. Not peer agents with a message bus, shared mutable state, or a mergeable state type. The signature is the architecture. A subagent returns text and nothing else -- no state update, no file handle, no mergeable value -- so a caller cannot wire one up as a concurrent writer even by accident. SubagentRequest carries only a role and a prompt, with no parent-state field by construction, so a subagent cannot reach the parent's conversation, files, or todos. This is what replaces Phase 1.6's CoW fork/merge + CRDT join. A CRDT can merge two edits to the same file without textual conflict; it cannot make the result coherent. Making the writer singular removes that entire failure class rather than managing it. Two roles, both read-only. may_write() and inherits_parent_context() are encoded as methods returning false rather than assumed, so adding a role that violates either is a visible decision rather than an oversight -- with tests asserting the invariant directly. Gatherer runs on the cheap tier by design: a frontier model in a read-only slot measured +0.4pp at 5.8x cost, corroborated by metaharness ADR-226's 5.4x null. Reviewer is documented as GATED rather than adopted, per ADR-278 §7, with ADR-226 named as the null it must beat. Summary returns are capped at the boundary rather than trusted: a subagent exists to reduce what reaches the parent's context, so one returning its whole transcript has inverted its own purpose. Tests: 182 core tests (was 174), including the single-writer invariant across all roles and the summary budget. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
The Evaluator is where all rvAgent meaning lands on the flywheel's four host-agnostic Score axes, which makes it the trust boundary for every downstream guarantee the gate provides. Rust side: RunOutcome is now serializable with the JS-side field names, since the Evaluator boundary is a process boundary rather than a function call. Adds item_id so an aggregator can detect a missing or duplicated item instead of silently averaging fewer runs, and a round-trip test pinning the wire names -- renaming one silently would make every aggregated Score wrong rather than failing loudly. JS side (scripts/rvagent-flywheel-evaluator.mjs): aggregation, lever validation, and an injected runItem so the seam is testable without spawning real agent runs. Three refusals are deliberate, each preventing a silently-wrong Score rather than a crash: - Zero runs score as maximally bad, never as a clean sweep. - A dropped item throws instead of shrinking the denominator, which would inflate every axis. - A policy naming an unapplied lever throws, because such a mutation produces a run identical to baseline that the optimizer reads as "no effect". COST_PER_WIN_NO_WINS is duplicated as Number.MAX_VALUE with the reason stated in both files: Infinity serializes to JSON null, and the gate reads null > n as false, so a zero-win policy would silently pass the cost clause. Verified the two implementations agree by running both over the same four cases -- mixed (3.25), success-but-noop, zero-wins, and empty -- rather than assuming a shared spec keeps them in sync. Tests: 19 policy tests, 184 core tests green. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
Answers two questions asked together. The answers turn out to be independent, and the second matters far more. C: no. Every capability commonly cited as C-only has a production Rust path in 2026 -- io_uring (pure-Rust crate), CUDA (cudarc dynamic loading), BLAS (faer matches or beats OpenBLAS), AVX-512 (stable since Rust 1.89, FP16 since 1.94). Two results are decisive rather than suggestive: zlib-rs is faster than zlib-ng in C AND is the fastest WASM zlib, and Qdrant -- the closest analogue -- spent two minor versions REMOVING its one C++ dependency, citing interop friction. The most-cited pro-C evidence does not survive decomposition. SimSIMD/NumKong's 20-118x headline compares its f16 against GCC's f32; there is no f32-vs-f32 row. Same-width it is 1.15-2.1x. PDX (SIGMOD '25) beat SimSIMD and FAISS hand-written kernels by 2.0x average using plain scalar C++ with no intrinsics, purely via data layout, and NumKong loses bulk scoring by 1.85-3.04x for want of a bulk API. Two expert C teams differ from each other more than Rust differs from C. Cost lands where RuVector is most exposed: 37 of 166 crates are WASM, and wasm32-unknown-unknown has no C/C++ toolchain by design. Every C feature would need reimplementing in Rust for the browser anyway. Miri also cannot execute across an FFI boundary, which for a concurrent index is a real loss. The one genuine gap -- f32 reduction reassociation, historically 8.4x -- closes in Rust 1.98 on 2026-08-20 via float_algebraic. What remains blocked is narrower than assumed: f16/bf16 arithmetic and ARM SVE/SME. The finding that matters more than C: ruvector-sota-bench measures against ann-benchmarks.com, which is deprecated and now redirects to VIBE, on SIFT/GloVe/Deep -- exactly the datasets VIBE exists because they are no longer representative. ADR-267 does not mention VIBE. Every SOTA claim we could make today rests on an unmaintained artifact. Same failure mode as the withdrawn SWE-bench-Verified gate in ADR-277. Recall@k is itself under credible attack: 1/Ratio@k reaches equal downstream quality with 1.86-9.36x fewer distance computations. Program, ranked: retarget to VIBE + 1/Ratio@k (blocks everything else); 8-bit rotational quantization (>99% recall10@10, 4x compression, zero training, days of work); SymphonyQG-class packed quantized graph; streaming-stable quantization where every incumbent is weak and which is our actual workload; adaptive filtered-query router; MUVERA FDE; CAGRA build via cuvs-sys. Records two corrections to my own earlier survey: an initial pass wrongly concluded there were no binary/hamming popcount kernels, having generalized from one file across a 166-crate workspace -- ruvector-rabitq/src/scan.rs has had AVX-512 VPOPCNTDQ all along. And the two research threads report TurboQuant against different baselines; the ADR states both rather than conflating them. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
…NOT adopted Implements the PDX (SIGMOD '25) vertical/dimension-major layout that ADR-279 5.1 ranked first in the kernel build order, benchmarks it against the existing row-major batch path, and records that it did not reproduce. Measured on this host (AVX-512, both paths built with target-cpu=native): PDX is 0.65-0.88x on cache-resident working sets (i.e. SLOWER) and only 1.09-1.29x when streaming. The paper reports ~2.0x. The first benchmark run was invalid and is worth recording. It showed PDX 14-18x slower, because the row-major path runtime-dispatches to AVX-512 via is_x86_feature_detected! while the new code compiled for baseline x86-64 with SSE2. That was an ISA comparison presented as a layout comparison. The corrected comparison is still confounded: the row-major path takes Vec<&[f32]> built from Vec<Vec<f32>>, so it pointer-chases across 4096 separate heap allocations, while PdxIndex is one contiguous buffer. The streaming win may be allocation contiguity rather than vertical layout. A clean experiment needs a contiguous row-major baseline. Reading: at f32 these workloads are bandwidth-bound, so layout cannot help much. That strengthens rather than weakens the case for 8-bit rotational quantization (ADR-279 5 item 2) -- 4x less data to stream attacks the actual bottleneck. Revisit PDX for quantized codes, where the working set may shrink enough to become compute-bound. The code is kept: it is correct, tested against a naive reference across block boundaries and padding lanes, and is the natural substrate for the quantized retry. It is not wired into any hot path. Also documents why the existing batch_euclidean is not a bulk kernel -- its TILE_SIZE chunking is decorative, since the inner call is identical to a flat loop over single-pair distances. Tests: 8 pdx tests including ragged-block rejection, oversized-block rejection, partial-block padding, and cross-block-boundary correctness at dim=129. Co-Authored-By: claude-flow <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
Co-Authored-By: claude-flow <ruv@ruv.net>
…, no-op HITL gate, Gemini request bugs Code-review fixes on top of the harness work: rvagent-tools (local.rs): - Reject dangling-symlink path components and writes through any symlink; refuse writes through hard links (nlink > 1, unix) — closes workspace write-escape; module doc now states exact confinement guarantees - Saturating offset+limit in read (u64::MAX no longer panics) - Char-boundary-safe output truncation - execute(): own process group, concurrent pipe draining (no >64KB deadlock), SIGKILL of the whole tree on timeout, child reaped - glob/grep no longer follow symlinked dirs (loop + read-escape fix) rvagent-middleware: - middleware_by_name takes PipelineConfig; HITL built from interrupt_on with a conservative default gate (shell/file mutation) shared by build_default_pipeline — both construction paths now match - HITL block message states reality (call dropped, how to configure) - Unknown middleware names are hard errors instead of silent skips - unicode_security/memory/skills/summarization by-name arms honor config - Retry backoff capped at 60s; summarization preview char-boundary-safe - masking truncate_tool_result respects caps smaller than the marker rvagent-backends: - Gemini: consecutive tool results grouped into one Content (parallel tool calls no longer 400); Candidate deserializes finishReason; missing candidates/content/parts produce descriptive errors instead of silent empty success; tool schemas sanitized to Gemini's OpenAPI subset (nullable arrays, format filtering, type-less fallbacks); parameterless tools omit parameters - Anthropic: consecutive tool results grouped into one user message rvagent-cli: - Pipeline built via validated by-name path; RVAGENT_AUTO_APPROVE=1 opt-out (stderr-visible) for unattended runs, fail-closed by default Co-Authored-By: claude-flow <ruv@ruv.net>
Owner
Author
Review + fix summary (automated multi-agent review)Full adversarial review of the ~9.5k-line diff found 3 blockers and ~15 majors in the new rvAgent code; all blockers and the tractable majors are fixed in Blockers fixed
Also fixedPanics on model-supplied input (read limit overflow, UTF-8 truncation in three places), execute() pipe deadlock >64KB + process-group kill + zombie reap, unknown middleware names now hard errors, sequential tool-call panic containment in graph.rs, max_iterations off-by-one, retry backoff cap. Known follow-ups (deliberately deferred)
🤖 Generated with claude-flow |
ruvnet
marked this pull request as ready for review
August 2, 2026 15:39
ruvnet
added a commit
that referenced
this pull request
Aug 2, 2026
Brings the ruvector npm package on main up to the 0.2.40 release content (published from an unmerged branch: metaharness SDK/CLI/MCP surface, ONNX embedder improvements, embedding provenance) and bumps to 0.2.41, published post-PR-#752 with rebuilt NAPI binaries on main. Co-Authored-By: claude-flow <ruv@ruv.net>
ruvnet
added a commit
that referenced
this pull request
Aug 2, 2026
…beddings, nightly research quality gate - ADR-280: RVF durable self-contained metadata (rvf-types metadata.rs, runtime options/store/filter/safety_net/vector_slab wiring, crash-safety and durability integration tests) - ADR-281: role-aware embedding APIs in ruvector-core (embeddings.rs, agenticdb.rs, error taxonomy across core/hailo/hailo-cluster, role test) - ADR-282: nightly research quality gate (research-gate scripts + schemas, research-* GitHub workflows, CODEOWNERS, sota-bench metaharness harness with scorePolicy + darwin.ts, gate override + promote pipelines) ADRs renumbered from 273-275 to 280-282 after PR #752 took 273-279. Co-Authored-By: claude-flow <ruv@ruv.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Research answering: how can rvagent be implemented more like the Hermes harness, integrated with ruvnet/metaharness and ruvnet/ruflo, to create the best SOTA harness?
Adds
docs/research/rvagent-hermes-harness/(4 docs, ~666 lines), synthesized from four parallel investigations: web research on NousResearch/hermes-agent, and deep code audits ofcrates/rvAgent/,ruvnet/metaharness, andruvnet/ruflo.Contents
.swarmsubstrate with ADR-323 provenance), trajectory→SKILL.md synthesis with Darwin/GEPA-style evolution via the policy-genome seam, real subagents + A2A as the federation layer, explicit optional-both-ways integration contracts (ADR-150/256 invariants).Notes
🤖 Generated with claude-flow
https://claude.ai/code/session_019mt39sZNUecTJBBBqPRH9F
Generated by Claude Code