fix(docs): correct a false halt claim; pin HOW the free path reaches VERIFIED - #298
Merged
Conversation
…ly reports `docs/EXECUTION_PROFILES.md` told readers that the tutorial's `?fault=optimistic` probe terminates `HALTED` / `HALTED_BEFORE_EFFECT` "with nothing persisted", and cited `tests/e2e/test_free_path_e2e.py` as the pin. That test pins `RECONCILIATION_REQUIRED`, and has since #280. The prose is what is wrong; the code is right, and deliberately so. #281 wrote the paragraph, #280 then made `HALTED_BEFORE_EFFECT` require positively established absence for every declared effect of every consequential step, flipped the test, and rewrote the taxonomy table in this same file -- but not this paragraph. The document has been self-contradictory since: line 77 claimed `HALTED_BEFORE_EFFECT` for a step that reached actuation, while line 99 said such a step "does not qualify". Measured on the real probe: the save declares 2 mined effects, the retained verifier returns evidence for 1 (`refuted`), and the store is in fact empty -- but only the test's out-of-band `GET /api/db` snapshot knows that. The runtime never held absence evidence for the second effect, so it reports `RECONCILIATION_REQUIRED`. This matters more than a stale line. `HALTED_BEFORE_EFFECT` tells an operator there is nothing to reconcile. Documenting it for a run that cannot support the claim is exactly the failure the transaction taxonomy exists to prevent, described in the file that defines the taxonomy. Rewrites the paragraph to state both probes' real outcome, why the optimistic case cannot claim absence, and that this is the taxonomy working rather than a gap in it.
`VERIFIED` is one bit, and one bit is a weak release gate. A run can end
`VERIFIED` for the wrong reason and every existing assertion in
`tests/e2e/test_free_path_e2e.py` stays green:
- the structural rung stops resolving and the ladder falls to OCR or
geometry -- weaker evidence, same verdict;
- an actuation path leaves the DOM for synthesised pointer events;
- the save loses its `irreversible` classification, and with it the armed
identity gate;
- a heal fires on an undrifted screen;
- the mined effect contract changes shape, so `VERIFIED` is asserted
against a different promise than the one that was reviewed.
Adds `tests/golden/tutorial_run.json`: the whole free path projected onto
a closed-vocabulary shape -- per step the resolution rung, actuation,
delivery status, identity status/mode, risk class, postcondition verdict,
heal flag and effect evidence; plus step count, `rung_counts`, identity
armed/applicable counts, required/passed contract counts, the mined
effect-contract hashes, model calls, cost, and the outcome fields. The
comparison walks both documents leaf by leaf and names the exact field
that moved rather than dumping two blobs.
Determinism: every pinned field was byte-identical across 8 local runs
under widely varying machine load. `TutorialResult.bundle_digest` is the
one unstable field (the bundle carries retained frames) and is
deliberately excluded. Nothing here is a timing, a path, or a temp dir.
Mutation proof. Drop the `field_equals` effect from
`compiler/effect_mining.py` -- the partial-save catch -- and the free path
still reaches `VERIFIED`, `test_free_path_terminates_verified_with_-
independent_effect_evidence` still passes, and this is what fails:
changed effects_confirmed: golden=2 now=1
changed required_contracts.effect: golden=2 now=1
the run no longer emits steps[5].effect_evidence[1].verdict
the run no longer emits effect_journal[0].intended_effect_contract_hashes[1]
Cost. Zero additional runner minutes. The assertion reads the report the
module-scoped `free_path` fixture already produces, so it adds a
comparison, not a run. The `e2e-browser` job now runs that module as its
FIRST step and `--ignore`s it from the directory run below, so the core
loop answers in the first ~2 minutes of a ~21-minute job instead of the
last -- ordering, not spend. No new workflow, no new job, no new required
context, and no change to any trigger.
Regenerate deliberately, never as a reflex:
OPENADAPT_UPDATE_GOLDEN=1 pytest tests/e2e/test_free_path_e2e.py
python scripts/check_release_consistency.py --write-public-artifact-inventory
Boundary: the artifact is enum names, key names, counts, and contract
hashes. No values, no record content, no screen text, no paths, no
tuning. Mechanism, not data -- the public side of AGENTS.md 4.2.
The adapter-result table already qualifies the `confirmed` row ("run-level
`VERIFIED` is decided by the run classifier") but not the `refuted` row,
so the table reads as though one refuted effect settles the run outcome.
It does not, and has not since #280: run-level `HALTED_BEFORE_EFFECT`
requires absence for every declared effect of every consequential step. The
tutorial's own `?fault=optimistic` probe is exactly this case -- one
refuted effect beside one unverified effect -- and terminates
`RECONCILIATION_REQUIRED`.
Adds the symmetric caveat.
abrichr
enabled auto-merge (squash)
July 28, 2026 03:06
abrichr
force-pushed
the
agent/halt-truth-and-golden
branch
from
July 28, 2026 03:06
ec4de8d to
55b1b73
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things, both about the same property: not asserting more than the evidence supports.
1. The false sentence
docs/EXECUTION_PROFILES.mdsaid:The test it names pins
RECONCILIATION_REQUIREDfor that fault, and has since #280.The prose is wrong. The code is right, and deliberately so. #281 (
2752813) wrote this paragraph. #280 (11c115c) then madeHALTED_BEFORE_EFFECTrequire positively established absence for every declared effect of every consequential step, flipped the test, and rewrote the taxonomy table twenty lines below in this same file — without touching the paragraph. The document has contradicted itself ever since: line 77 claimedHALTED_BEFORE_EFFECTfor a step that reached actuation, line 99 said such a step "does not qualify".Measured on the real probe (
?fault=optimistic, tutorial bundle,RestRecordVerifieron/api/db):The store really is empty. Only the test's out-of-band snapshot knows that. The runtime held absence evidence for one of two declared effects, so
RECONCILIATION_REQUIREDis the strongest true thing it can say.Why this is not a typo.
HALTED_BEFORE_EFFECTis the outcome that tells an operator there is nothing to reconcile. Documenting it for a run that cannot support the claim is precisely the failure the transaction taxonomy exists to prevent — written into the file that defines the taxonomy. Fixed in prose; no behaviour changes.2. The golden task
VERIFIEDis one bit, and one bit is a weak release gate. A run can reach it for the wrong reason with every existing assertion in this module still green: the structural rung degrading to OCR or geometry, actuation leaving the DOM, the save losing itsirreversibleclass and with it the armed identity gate, a heal firing on an undrifted screen, or the mined effect contract changing shape soVERIFIEDmeans something different than it did at review.Candidate.
openadapt-flow tutorial— record -> compile -> certify -> run under Standard against an independent/api/dboracle -> receipt. Suitable: hermetic (loopback only), zero model calls, ~14.5s locally end to end (the ~4.8s figure is the run stage alone), and the module fixture already exists.What it asserts.
tests/golden/tutorial_run.json, compared leaf by leaf:execution_outcome,transaction_outcome,execution_profile, billable, success,production_eligible,haltmodel_calls(0),est_model_cost_usd,heal_counteffect_tier(1 = INDEPENDENT_SYSTEM), governed minimum tier, policy, record count, required/passed contract counts, the mined effect-contract hashesrung_counts, identity armed/applicable, and per step: rung, actuation, delivery status, identity status/mode, risk class, postcondition verdict, heal flag, effect evidenceA failure names the field, not two documents.
Determinism. Byte-identical across 8 local runs under widely varying machine load (11s to 229s wall clock, same shape).
bundle_digestis the one unstable field — the bundle carries retained frames — and is excluded. Nothing pinned is a timing, path, or temp dir.Mutation proof. Drop the
field_equalseffect incompiler/effect_mining.py(the partial-save catch). The free path still reachesVERIFIED;test_free_path_terminates_verified_with_independent_effect_evidencestill passes; this is what fails:Restored;
11 passed.Cost and trigger. Zero additional runner minutes. No new workflow, no new job, no new required context, no trigger change. The assertion reads the report the module-scoped
free_pathfixture already produces, so it is a comparison rather than a run.e2e-browsernow runs that module as its FIRST step and--ignores it from the directory run below: the core loop answers in the first ~2 minutes of a ~21-minute job instead of the last. Ordering, not spend.A dedicated job was considered and rejected: it would re-pay checkout +
pip install -e .[dev]+ Playwright (~2.5-3.5 min on every push, on bothpull_requestandpush: main) to duplicate an installe2e-browseralready has — against AGENTS.md 5.1's "run one authoritative check per contract, reuse its artifact downstream".Regenerating — deliberately, never as a reflex:
The second command is required because the artifact is inventoried in
public-artifacts.json, so the shape cannot move without a reviewed hash change.Boundary. Enum names, key names, counts, and contract hashes. No values, no record content, no screen text, no paths, no tuning. Mechanism, not data — the public side of AGENTS.md 4.2.
Verified locally
ruff check openadapt_flow·ruff format --check·mypy(218 files) ·scripts/check_consistency.py·scripts/check_release_consistency.py·scripts/check_bundle_phi.py·pytest tests/e2e/test_free_path_e2e.py(11 passed).Noted, not fixed here
Adjacent halt-prose findings from the same sweep, each in another lane's files:
openadapt-maintenancedocs/concepts/durable-runtime.md:21-23— "When the run halts (an unhandled screen, a refuted write, an escalation) ... nothing after the checkpoint was performed." False for exactly the case it names: a refuted write was delivered, andtransaction.py:362-370says so explicitly ("a postcondition is checked after the click, so an over-halt ... is proof the write was already delivered"). Separate PR.openadapt-desktopdocs/DECISION_PORTAL.md:134-141describes arejectaction the engine never offers (attended.py:168has four). feat(attended): let an operator reject a halt and end the run #295 is implementing it — left alone.🤖 Generated with Claude Code
https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM