You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Proposal: configurable ODD evaluation with automatic execution graphs
#5846
Design proposal for discussion. This requests architectural feedback before proposing implementation PRs. It does not claim maintainer approval, a finished evaluation platform, or demonstrated advantages of ODD/RDD.
I would like feedback on an evaluation lab for Gentle AI that makes agent configurations comparable through reproducible execution, requirement-level evidence, and automatically generated execution graphs. The goal is not another fixed leaderboard or a benchmark designed to show that a particular workflow wins.
The main question: does this belong as an opt-in experimental lab in Gentle AI, with a small ODD adapter and reusable evaluation core, or should it live separately?
Why this is useful
A final pass/fail score does not explain how an agent reached its result. We also need to inspect retries, corrections, delegation, memory interactions, parallel work and resource consumption, without assuming that a longer or more elaborate route is better.
Examples of questions the system should support:
Does a model/effort change improve acceptance at a comparable cost?
Does prepared memory help this task population compared with a verified empty store?
Does a harness patch or a Gentle AI release change routing, latency or residual errors?
Later, does RDD reduce independently confirmed residual defects, and at what total review/refutation/correction cost?
These are possible studies, not hypotheses already confirmed. Tasks, configurations, evaluators and metrics must be replaceable. A simple evaluation should not require a causal study.
Proposed architecture
flowchart TB
P[Versioned experiment plan] --> V[Validate declared configuration]
T[Task and initial-state fixtures] --> R[Authorized isolated attempt runner]
C[Immutable configuration and component identities] --> R
V --> R
R --> E[Captured events and artifacts]
R --> Q[Independent task-specific evaluator]
E --> S[Evidence records with coverage and source references]
Q --> S
S --> G[Deterministic execution graph derivation]
G --> A[Versioned cross-attempt alignment]
S --> D[Descriptive outcome and resource report]
A --> D
D --> H[Human inspection and next experiment]
X[Separately specified causal study] -.-> D
Loading
This is a proposed data flow, not an implemented orchestration API. Configuration validation is not execution authorization. Each boundary should preserve the ordinary consent, security and review requirements of the host.
Observable tool/model/delegation/artifact events and provenance. Explicitly report unavailable instrumentation.
Evidence core
Preserve identities, failures, missingness, resource semantics and references to original observations.
Graph/comparison layer
Derive graphs reproducibly, retain original attempts and expose alignment uncertainty.
Analysis layer
Descriptive comparisons first; a separately justified causal analysis only where the study supports it.
Record native Gentle AI, host/harness, SDK, Engram, model/effort, relevant tools, task/evaluator versions and source/build digests separately. A moving branch name is not an immutable treatment identity.
An ODD adapter observes organic routing. It must not force an explorer-worker-verifier pipeline. Inline work, parallel branches and revisited phases are all valid observations.
Automatic graphs are a core output
The graph should be generated from captured evidence, not manually drawn after the fact or reconstructed from model explanations. It should remain linked to the task, configuration/treatment assignment, artifacts, checks and measured outcomes.
Example execution graph
The following is a synthetic illustration, not a captured run or required ODD route:
flowchart LR
I[Initial fixture] --> E[Explore occurrence 1]
E --> W[Implement occurrence 1]
W --> P[Patch artifact v1]
P --> T[Test occurrence 1: fail]
T --> C[Correct occurrence 2]
P --> C
C --> P2[Patch artifact v2]
P2 --> T2[Test occurrence 2: pass]
P2 --> V[Independent evaluation]
T2 --> O[Attempt outcome]
V --> O
Loading
Every edge needs an observed dependency/data-flow relation or an explicit derivation rule with source references. Timestamp order alone is insufficient. Repeated work gets distinct occurrence and artifact-version nodes, so corrections do not produce artificial cycles.
Failed and partial attempts remain visible. Missing links are unknown, not invented. We do not capture private chain-of-thought or infer hidden routing decisions.
Comparing two attempts without manufacturing a combined execution
Solid edges belong to the individual executions. Dotted links are cross-attempt correspondences under a versioned alignment policy, not dependencies or causal effects. Ambiguous and unmatched nodes must survive alignment. The same evidence and derivation/alignment versions should reproduce the same graph without model calls.
Execution tracing is not causal inference
A provenance graph answers what depended on what in this execution. A causal hypothesis graph expresses assumptions about what an intervention could change. They must not be conflated.
For example, the following is a study hypothesis, not an inferred result:
flowchart LR
D[Task difficulty] --> M[Observed routing and corrections]
D --> Y[Quality and total resource outcome]
H[Assigned harness configuration] --> M
H --> Y
M --> Y
Loading
Task difficulty can affect both route and outcome. Routing can itself be affected by the configuration, so controlling for it mechanically may obscure the total effect we wanted to estimate.
A supported causal claim would need a declared intervention/contrast, assignment unit, comparable initial state, replication/dependence handling and an appropriate uncertainty analysis. Repeated attempts of one task are not independent tasks. A release-versus-release comparison that changes several components is a bundle contrast, not proof about one component.
No estimator or automatic causal discovery method is selected by this proposal.
Measurement rules
Freeze requirements and evaluators before judging outputs; report every criterion as pass, fail or unknown with evidence.
Retain failed, cancelled, rejected and partial attempts. Declare denominators and inclusion rules rather than ranking only successful runs.
Keep provider-reported token categories, estimated prices and actual billed cost separate. Unknown usage is not zero.
Avoid double-counting inclusive parent usage and child usage. Parallel child durations do not add up to elapsed wall time.
Distinguish absent, disabled, verified-empty, prepared and retained memory. Availability or retrieval alone does not prove correct use.
Include preparation/reset and, in future RDD studies, review, refutation, corrections and validation under explicit accounting boundaries.
Evaluate confirmed defects and residual errors independently. Review findings are not automatically true positives; disagreement and unavailable adjudication remain explicit.
What exists today, and what does not
This is a local prototype, not a published or merge-ready submission. The supporting work currently lives under experiments/handoff-lab/.
Area
Actual status
Protocol design
Written in protocol.md, including state, identity, counting and comparison boundaries.
Offline plan validation
Implemented in protocol/validate.mjs with tests; validates declared structure, not live execution.
Offline evidence ingestion
Existing evidence contract and ingestion checks in data/; supplied records are not proof of capture completeness.
Isolated SDK startup/memory smoke
A narrow credential-free baseline was observed; not model-driven ODD compliance or an experiment.
Post-correction real Docker verification returned create_failed and cleanup unknown on its first scenario. Later scenarios and targeted independent clearance remain pending. Earlier pre-correction runtime passes do not clear this candidate.
Two-attempt file/Engram freshness
Planned, not established by the new runner.
Live agent event capture, automatic graphs and analysis
Proposed, not implemented. No measured ODD/RDD benefit is claimed.
The deterministic suite is not a substitute for actual daemon/container evidence or independent verification. No paid experimental model study has been run.
Suggested delivery, subject to maintainer approval
Agree on scope and repository placement here, then open an implementation issue for approval.
Propose the protocol/evidence/validator foundation in focused review slices.
Complete and independently verify runtime ownership, limits and fresh-state controls before delivering that runner.
Add capture and a deterministic graph builder with known-answer synthetic cases: inline execution, parallel branches/joins, corrections, missing parents and failed attempts.
Add versioned cross-attempt alignment and descriptive reports, preserving unmatched observations.
Select a small consented ODD study with explicit task/evaluator/model budgets. Consider RDD and causal estimators only after the necessary evidence and study design exist.
Each implementation PR should reference an approved issue and follow the project's review budget of at most 400 additions plus deletions, unless maintainers approve an exception. Tests and docs stay with their behavior. This discussion does not request approval to merge unfinished runtime work or weaken ordinary review/consent requirements.
Papers and standards informing the design
These references motivate individual design choices. They do not validate this prototype or prescribe a dependency stack.
Chiang et al. (2024), Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, ICML.Paper and proceedings. Supports pairwise human-preference evaluation and statistical ranking as useful comparison patterns. It does not establish automatic agent execution DAGs or causal tracing. A preference evaluation would be an optional evaluator, not a replacement for task-specific correctness checks.
Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, ICLR 2024.Paper. Motivates evaluating software agents against repository-level issue resolution rather than fluent output alone. SWE-bench is a possible task-adapter reference, not the selected corpus or proof that its setup transfers unchanged to ODD.
W3C (2013), PROV-DM: The PROV Data Model.Recommendation. Provides vocabulary for entities, activities, agents, usage, generation and derivation. It informs provenance modeling, not a claim that every PROV graph is acyclic or that provenance proves causality. Our execution DAG would be a constrained, occurrence/version-aware projection.
Hernán and Robins (2020), Causal Inference: What If.Authors' book and materials. A methodological reference for separating causal questions and assumptions from observational associations. This is a book, not an agent benchmark or an estimator selected for the lab.
Source checks for this draft covered the Arena proceedings abstract, SWE-bench abstract, relevant PROV-DM passages and the authors' book page. This is not a systematic literature review or a full audit of those implementations.
Questions for maintainers
Is an opt-in lab under experiments/ appropriate, or should the reusable core live in a separate repository?
Does the distinction between evidence-backed execution graphs, cross-attempt alignment and separately justified causal studies fit the project's direction?
Would you prefer reviewing the protocol/validator foundation first, before any live capture or rendering choice?
Material AI assistance (el Gentleman/Pi and subagents) was used for design drafting, reference investigation and local prototype development. Verification performed so far and its limitations are described above; this does not imply maintainer approval.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Design proposal for discussion. This requests architectural feedback before proposing implementation PRs. It does not claim maintainer approval, a finished evaluation platform, or demonstrated advantages of ODD/RDD.
I would like feedback on an evaluation lab for Gentle AI that makes agent configurations comparable through reproducible execution, requirement-level evidence, and automatically generated execution graphs. The goal is not another fixed leaderboard or a benchmark designed to show that a particular workflow wins.
The main question: does this belong as an opt-in experimental lab in Gentle AI, with a small ODD adapter and reusable evaluation core, or should it live separately?
Why this is useful
A final pass/fail score does not explain how an agent reached its result. We also need to inspect retries, corrections, delegation, memory interactions, parallel work and resource consumption, without assuming that a longer or more elaborate route is better.
Examples of questions the system should support:
These are possible studies, not hypotheses already confirmed. Tasks, configurations, evaluators and metrics must be replaceable. A simple evaluation should not require a causal study.
Proposed architecture
This is a proposed data flow, not an implemented orchestration API. Configuration validation is not execution authorization. Each boundary should preserve the ordinary consent, security and review requirements of the host.
Record native Gentle AI, host/harness, SDK, Engram, model/effort, relevant tools, task/evaluator versions and source/build digests separately. A moving branch name is not an immutable treatment identity.
An ODD adapter observes organic routing. It must not force an explorer-worker-verifier pipeline. Inline work, parallel branches and revisited phases are all valid observations.
Automatic graphs are a core output
The graph should be generated from captured evidence, not manually drawn after the fact or reconstructed from model explanations. It should remain linked to the task, configuration/treatment assignment, artifacts, checks and measured outcomes.
Example execution graph
The following is a synthetic illustration, not a captured run or required ODD route:
Every edge needs an observed dependency/data-flow relation or an explicit derivation rule with source references. Timestamp order alone is insufficient. Repeated work gets distinct occurrence and artifact-version nodes, so corrections do not produce artificial cycles.
Failed and partial attempts remain visible. Missing links are unknown, not invented. We do not capture private chain-of-thought or infer hidden routing decisions.
Comparing two attempts without manufacturing a combined execution
flowchart TB subgraph A[Attempt A: configuration A] A1[Requirement R1] --> A2[Patch A] --> A3[Check A] end subgraph B[Attempt B: configuration B] B1[Requirement R1] --> B2[Patch B] --> B3[Check B] end A1 -. comparable requirement .-> B1 A3 -. comparable evaluator .-> B3Solid edges belong to the individual executions. Dotted links are cross-attempt correspondences under a versioned alignment policy, not dependencies or causal effects. Ambiguous and unmatched nodes must survive alignment. The same evidence and derivation/alignment versions should reproduce the same graph without model calls.
Execution tracing is not causal inference
A provenance graph answers what depended on what in this execution. A causal hypothesis graph expresses assumptions about what an intervention could change. They must not be conflated.
For example, the following is a study hypothesis, not an inferred result:
Task difficulty can affect both route and outcome. Routing can itself be affected by the configuration, so controlling for it mechanically may obscure the total effect we wanted to estimate.
A supported causal claim would need a declared intervention/contrast, assignment unit, comparable initial state, replication/dependence handling and an appropriate uncertainty analysis. Repeated attempts of one task are not independent tasks. A release-versus-release comparison that changes several components is a bundle contrast, not proof about one component.
No estimator or automatic causal discovery method is selected by this proposal.
Measurement rules
What exists today, and what does not
This is a local prototype, not a published or merge-ready submission. The supporting work currently lives under
experiments/handoff-lab/.protocol.md, including state, identity, counting and comparison boundaries.protocol/validate.mjswith tests; validates declared structure, not live execution.data/; supplied records are not proof of capture completeness.create_failedand cleanupunknownon its first scenario. Later scenarios and targeted independent clearance remain pending. Earlier pre-correction runtime passes do not clear this candidate.The deterministic suite is not a substitute for actual daemon/container evidence or independent verification. No paid experimental model study has been run.
Suggested delivery, subject to maintainer approval
Each implementation PR should reference an approved issue and follow the project's review budget of at most 400 additions plus deletions, unless maintainers approve an exception. Tests and docs stay with their behavior. This discussion does not request approval to merge unfinished runtime work or weaken ordinary review/consent requirements.
Papers and standards informing the design
These references motivate individual design choices. They do not validate this prototype or prescribe a dependency stack.
Source checks for this draft covered the Arena proceedings abstract, SWE-bench abstract, relevant PROV-DM passages and the authors' book page. This is not a systematic literature review or a full audit of those implementations.
Questions for maintainers
experiments/appropriate, or should the reusable core live in a separate repository?Material AI assistance (el Gentleman/Pi and subagents) was used for design drafting, reference investigation and local prototype development. Verification performed so far and its limitations are described above; this does not imply maintainer approval.
All reactions