[codex] PR1: record evidence-backed harness hypotheses - #34
Merged
Conversation
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 13 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
divo12
force-pushed
the
stack/v2-01-experiment
branch
from
September 3, 2026 07:49
dd67464 to
d6d19a7
Compare
There was a problem hiding this comment.
6 issues found across 15 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/ofw/preparation/policy.py">
<violation number="1" location="src/ofw/preparation/policy.py:57">
P1: Existing policy snapshots become unloadable after this change. Missing `baseline_reused` defaults to `False`, but `validate_controls_digest()` recomputes the digest with the new field and rejects the old digest; preserve legacy canonicalization or version and migrate the policy schema before deployment.</violation>
</file>
<file name="tests/test_hypothesis.py">
<violation number="1" location="tests/test_hypothesis.py:205">
P2: The new fixture supplies prediction fields, but the main successful recording test never verifies that either field reaches the receipt or persisted hypothesis. Add assertions for both fields (and, preferably, the serialized artifact) so this change cannot regress silently.</violation>
</file>
<file name="plugins/openflywheel/skills/workspace-init/SKILL.md">
<violation number="1" location="plugins/openflywheel/skills/workspace-init/SKILL.md:28">
P2: Because `experiment_id` is collected only in step 4, the assistant cannot evaluate this condition while following the required step-3 ordering. Defer the check until after collecting the ID and specify that declining adoption requires a new ID; otherwise `prepare_workspace` rejects the existing job with `launch_failed`.</violation>
</file>
<file name="src/ofw/preparation/service.py">
<violation number="1" location="src/ofw/preparation/service.py:204">
P2: When a reused baseline is already READY, a repeat `prepare` call re-runs `summarize` instead of returning the persisted observation. Re-adopt only while the state is RUNNING so identical retries remain stable.</violation>
</file>
<file name="src/ofw/evolution/hypothesis.py">
<violation number="1" location="src/ofw/evolution/hypothesis.py:416">
P2: `record_hypothesis` accepts repeated IDs within `predicted_task_ids` or `at_risk_task_ids` and persists them as separate entries. Reject duplicates within each list as well as overlap so task predictions remain a set-like, bounded contract.</violation>
<violation number="2" location="src/ofw/evolution/hypothesis.py:512">
P2: Reordering the same predicted or at-risk task IDs changes `HypothesisId`, so semantically identical hypotheses are not idempotent. Canonicalize both task lists after uniqueness validation and reuse those canonical tuples in the content, artifact, and observation.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| quality_target: NormalizedScore | ||
| max_iterations: IterationCount | ||
| no_improvement_limit: IterationCount | ||
| baseline_reused: bool = False |
There was a problem hiding this comment.
P1: Existing policy snapshots become unloadable after this change. Missing baseline_reused defaults to False, but validate_controls_digest() recomputes the digest with the new field and rejects the old digest; preserve legacy canonicalization or version and migrate the policy schema before deployment.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/ofw/preparation/policy.py, line 57:
<comment>Existing policy snapshots become unloadable after this change. Missing `baseline_reused` defaults to `False`, but `validate_controls_digest()` recomputes the digest with the new field and rejects the old digest; preserve legacy canonicalization or version and migrate the policy schema before deployment.</comment>
<file context>
@@ -54,6 +54,7 @@ class _ExperimentPolicyContent(StrictModel):
quality_target: NormalizedScore
max_iterations: IterationCount
no_improvement_limit: IterationCount
+ baseline_reused: bool = False
max_cost_per_task_usd: PositiveMetric | None = None
max_latency_seconds: PositiveMetric | None = None
</file context>
| source_commit=commit, | ||
| curation_id=curation_id, | ||
| curation_group_id=curation_group_id, | ||
| predicted_task_ids=("task-1", "task-2"), |
There was a problem hiding this comment.
P2: The new fixture supplies prediction fields, but the main successful recording test never verifies that either field reaches the receipt or persisted hypothesis. Add assertions for both fields (and, preferably, the serialized artifact) so this change cannot regress silently.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/test_hypothesis.py, line 205:
<comment>The new fixture supplies prediction fields, but the main successful recording test never verifies that either field reaches the receipt or persisted hypothesis. Add assertions for both fields (and, preferably, the serialized artifact) so this change cannot regress silently.</comment>
<file context>
@@ -202,6 +202,8 @@ def _request(
source_commit=commit,
curation_id=curation_id,
curation_group_id=curation_group_id,
+ predicted_task_ids=("task-1", "task-2"),
+ at_risk_task_ids=("task-3",),
patterns=(
</file context>
| 2. Optimization goal, primary metric, target, and stopping condition. Keep quality, cost, | ||
| and latency constraints separate rather than hiding them in one average. | ||
| 3. ITSM-bench root, Harbor executable, Harbor configuration, and expected task count. | ||
| 3. ITSM-bench root, Harbor executable, Harbor configuration, and expected task count. If an |
There was a problem hiding this comment.
P2: Because experiment_id is collected only in step 4, the assistant cannot evaluate this condition while following the required step-3 ordering. Defer the check until after collecting the ID and specify that declining adoption requires a new ID; otherwise prepare_workspace rejects the existing job with launch_failed.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At plugins/openflywheel/skills/workspace-init/SKILL.md, line 28:
<comment>Because `experiment_id` is collected only in step 4, the assistant cannot evaluate this condition while following the required step-3 ordering. Defer the check until after collecting the ID and specify that declining adoption requires a new ID; otherwise `prepare_workspace` rejects the existing job with `launch_failed`.</comment>
<file context>
@@ -25,7 +25,10 @@ when the evidence supports one. Collect, in order:
2. Optimization goal, primary metric, target, and stopping condition. Keep quality, cost,
and latency constraints separate rather than hiding them in one average.
-3. ITSM-bench root, Harbor executable, Harbor configuration, and expected task count.
+3. ITSM-bench root, Harbor executable, Harbor configuration, and expected task count. If an
+ exact terminal Harbor job already exists at `benchmark_root/jobs/<experiment-id>`, ask whether
+ to adopt it; set `reuse_existing_baseline=true` only after explicit confirmation so policy is
</file context>
| terminal = _terminal_observation(request, state) | ||
| if terminal is not None: | ||
| return terminal | ||
| if state.baseline_reused: |
There was a problem hiding this comment.
P2: When a reused baseline is already READY, a repeat prepare call re-runs summarize instead of returning the persisted observation. Re-adopt only while the state is RUNNING so identical retries remain stable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/ofw/preparation/service.py, line 204:
<comment>When a reused baseline is already READY, a repeat `prepare` call re-runs `summarize` instead of returning the persisted observation. Re-adopt only while the state is RUNNING so identical retries remain stable.</comment>
<file context>
@@ -176,9 +201,19 @@ def _poll(
PreparationErrorCode.REQUEST_CONFLICT,
request.experiment_id,
)
+ if state.baseline_reused:
+ return self._adopt_existing(request, state_directory, state)
continuation = self._continue_existing(request, state_directory, state)
</file context>
Suggested change
| if state.baseline_reused: | |
| if state.baseline_reused and state.phase is PreparationPhase.RUNNING: |
| source_commit=request.source_commit, | ||
| curation_id=request.curation_id, | ||
| curation_group_id=request.curation_group_id, | ||
| predicted_task_ids=request.predicted_task_ids, |
There was a problem hiding this comment.
P2: Reordering the same predicted or at-risk task IDs changes HypothesisId, so semantically identical hypotheses are not idempotent. Canonicalize both task lists after uniqueness validation and reuse those canonical tuples in the content, artifact, and observation.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/ofw/evolution/hypothesis.py, line 512:
<comment>Reordering the same predicted or at-risk task IDs changes `HypothesisId`, so semantically identical hypotheses are not idempotent. Canonicalize both task lists after uniqueness validation and reuse those canonical tuples in the content, artifact, and observation.</comment>
<file context>
@@ -479,6 +509,8 @@ def _content(
source_commit=request.source_commit,
curation_id=request.curation_id,
curation_group_id=request.curation_group_id,
+ predicted_task_ids=request.predicted_task_ids,
+ at_risk_task_ids=request.at_risk_task_ids,
patterns=tuple(
</file context>
| predicted: tuple[str, ...], | ||
| at_risk: tuple[str, ...], | ||
| ) -> None: | ||
| if set(predicted) & set(at_risk): |
There was a problem hiding this comment.
P2: record_hypothesis accepts repeated IDs within predicted_task_ids or at_risk_task_ids and persists them as separate entries. Reject duplicates within each list as well as overlap so task predictions remain a set-like, bounded contract.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/ofw/evolution/hypothesis.py, line 416:
<comment>`record_hypothesis` accepts repeated IDs within `predicted_task_ids` or `at_risk_task_ids` and persists them as separate entries. Reject duplicates within each list as well as overlap so task predictions remain a set-like, bounded contract.</comment>
<file context>
@@ -389,6 +409,14 @@ def _curated_artifact_ids(group: FailureGroupArtifact) -> tuple[str, ...]:
+ predicted: tuple[str, ...],
+ at_risk: tuple[str, ...],
+) -> None:
+ if set(predicted) & set(at_risk):
+ raise ValueError("predicted_task_ids and at_risk_task_ids must be disjoint")
+
</file context>
Suggested change
| if set(predicted) & set(at_risk): | |
| if ( | |
| len(set(predicted)) != len(predicted) | |
| or len(set(at_risk)) != len(at_risk) | |
| or set(predicted) & set(at_risk) | |
| ): |
feat: execute isolated candidates
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR1 makes the prepared experiment the sole authority boundary and records one evidence-backed harness hypothesis before any candidate edit.
ExperimentPolicySnapshotfrom validatedPrepareWorkspaceInput, Harbor configuration, and Git preparation results;HypothesisServicere-runs the existing exactFailurePatternMiningServiceover at most 50 explicitly supplied diagnosis IDs, rejects incomplete or misassigned evidence, validates the prepared branch/commit/clean state, and authorizes only exact policy editable paths;record_hypothesis,$hypothesis-former, package exports, and generatedPROGRAM.mdstop at the stable receipt and expose no candidate-editing path.Cause and user effect
The prior preparation output committed a human-readable YAML file but had no typed internal authority snapshot and no service after exact failure-pattern mining. An agent could describe a hypothesis, but the system could not prove that its evidence partition was exact, its source commit was current, or its targets were actually editable. This change makes those checks deterministic and fail-closed before edits.
Data flow and trust seams
Langfuse remains the trace source of truth. Hypothesis recording performs no provider I/O and stores no trace input, output, span, or provider payload. Git is local authority for branch, commit, and tracked cleanliness; the policy exact editable allowlist is the only path authorization source.
ComponentKindis taxonomy only.Immutable contracts
HypothesisIdand byte-identical receipt.Approved plan mutation
freshcommit8416e24781d5332cb40e5a7ef8cb14254e7c3905, preserving the base-owned 0.7.1 release/runtime pin. The System Book deletion is inherited fromfreshand absent from this PR's three-dot content diff.1d91d03402499377a6de97aff5e81323d1dc0747removedsrc/ofw/harness.py,src/ofw/runtime.py,HarnessRevision, asset ownership/access, E2B/canary, and scenario/install planes.editable_pathsreplaces component ownership/access authorization. No removed module or downstream interface is recreated.policy.jsonfail withpolicy_snapshot_requiredand must be re-prepared under a new experiment ID; YAML is not loosely migrated. Failure diagnoses now use schema v2 with a verified semantic content digest, so earlier diagnosis files must be recorded again before hypothesis formation.Local verification (not CI)
uv sync --extra dev --extra pluginuv run ruff check src tests plugins/openflywheel/scripts/mcp_server.py— passeduv run mypy src tests plugins/openflywheel/scripts/mcp_server.py— passed, 50 filesuv run pytest --cov=ofw --cov-report=term-missing --cov-fail-under=90 -q— 236 passed, 2 opt-in live tests skipped, 95.35% project coveragequick_validate.py— trace-query-planner, outcome-recorder, failure-miner, failure-pattern-miner, failure-curator, workspace-init, and hypothesis-former all passedgit diff --check, excluded-artifact scan, secret scan, base/merge-base verification, and status audit passedNon-goals
No harness/runtime resurrection, generic live-agent adapter, candidate workspace, editor, runner, gate, controller, publication, release, scenario, install, or live-loop interface is included. No dependency was added, and no Langfuse collection, storage, judging, or benchmark behavior changed.