Skip to content

fix(docs): correct a false halt claim; pin HOW the free path reaches VERIFIED - #298

Merged
abrichr merged 3 commits into
mainfrom
agent/halt-truth-and-golden
Jul 28, 2026
Merged

fix(docs): correct a false halt claim; pin HOW the free path reaches VERIFIED#298
abrichr merged 3 commits into
mainfrom
agent/halt-truth-and-golden

Conversation

@abrichr

@abrichr abrichr commented Jul 28, 2026

Copy link
Copy Markdown
Member

Two things, both about the same property: not asserting more than the evidence supports.

1. The false sentence

docs/EXECUTION_PROFILES.md said:

With ?fault=optimistic (the screen says Saved, the server rejected the write) the run is HALTED / HALTED_BEFORE_EFFECT with nothing persisted; ... Both are pinned in tests/e2e/test_free_path_e2e.py.

The test it names pins RECONCILIATION_REQUIRED for that fault, and has since #280.

The prose is wrong. The code is right, and deliberately so. #281 (2752813) wrote this paragraph. #280 (11c115c) then made HALTED_BEFORE_EFFECT require positively established absence for every declared effect of every consequential step, flipped the test, and rewrote the taxonomy table twenty lines below in this same file — without touching the paragraph. The document has contradicted itself ever since: line 77 claimed HALTED_BEFORE_EFFECT for a step that reached actuation, line 99 said such a step "does not qualify".

Measured on the real probe (?fault=optimistic, tutorial bundle, RestRecordVerifier on /api/db):

OUT HALTED RECONCILIATION_REQUIRED
declared_effect_contract_hashes 2
effect_evidence [('refuted', 'rest')]
db records []

The store really is empty. Only the test's out-of-band snapshot knows that. The runtime held absence evidence for one of two declared effects, so RECONCILIATION_REQUIRED is the strongest true thing it can say.

Why this is not a typo. HALTED_BEFORE_EFFECT is the outcome that tells an operator there is nothing to reconcile. Documenting it for a run that cannot support the claim is precisely the failure the transaction taxonomy exists to prevent — written into the file that defines the taxonomy. Fixed in prose; no behaviour changes.

2. The golden task

VERIFIED is one bit, and one bit is a weak release gate. A run can reach it for the wrong reason with every existing assertion in this module still green: the structural rung degrading to OCR or geometry, actuation leaving the DOM, the save losing its irreversible class and with it the armed identity gate, a heal firing on an undrifted screen, or the mined effect contract changing shape so VERIFIED means something different than it did at review.

Candidate. openadapt-flow tutorial — record -> compile -> certify -> run under Standard against an independent /api/db oracle -> receipt. Suitable: hermetic (loopback only), zero model calls, ~14.5s locally end to end (the ~4.8s figure is the run stage alone), and the module fixture already exists.

What it asserts. tests/golden/tutorial_run.json, compared leaf by leaf:

outcome execution_outcome, transaction_outcome, execution_profile, billable, success, production_eligible, halt
cost model_calls (0), est_model_cost_usd, heal_count
evidence effects required/confirmed, effect_tier (1 = INDEPENDENT_SYSTEM), governed minimum tier, policy, record count, required/passed contract counts, the mined effect-contract hashes
mechanism step count, rung_counts, identity armed/applicable, and per step: rung, actuation, delivery status, identity status/mode, risk class, postcondition verdict, heal flag, effect evidence

A failure names the field, not two documents.

Determinism. Byte-identical across 8 local runs under widely varying machine load (11s to 229s wall clock, same shape). bundle_digest is the one unstable field — the bundle carries retained frames — and is excluded. Nothing pinned is a timing, path, or temp dir.

Mutation proof. Drop the field_equals effect in compiler/effect_mining.py (the partial-save catch). The free path still reaches VERIFIED; test_free_path_terminates_verified_with_independent_effect_evidence still passes; this is what fails:

E  AssertionError: the bundled tutorial no longer runs the way the golden artifact records.
E    changed effects_confirmed: golden=2 now=1
E    changed effects_required: golden=2 now=1
E    changed passed_contracts.effect: golden=2 now=1
E    changed required_contracts.effect: golden=2 now=1
E    the run no longer emits effect_journal[0].intended_effect_contract_hashes[1]
E    the run no longer emits steps[5].effect_evidence[1].verdict

Restored; 11 passed.

Cost and trigger. Zero additional runner minutes. No new workflow, no new job, no new required context, no trigger change. The assertion reads the report the module-scoped free_path fixture already produces, so it is a comparison rather than a run. e2e-browser now runs that module as its FIRST step and --ignores it from the directory run below: the core loop answers in the first ~2 minutes of a ~21-minute job instead of the last. Ordering, not spend.

A dedicated job was considered and rejected: it would re-pay checkout + pip install -e .[dev] + Playwright (~2.5-3.5 min on every push, on both pull_request and push: main) to duplicate an install e2e-browser already has — against AGENTS.md 5.1's "run one authoritative check per contract, reuse its artifact downstream".

Regenerating — deliberately, never as a reflex:

OPENADAPT_UPDATE_GOLDEN=1 pytest tests/e2e/test_free_path_e2e.py
python scripts/check_release_consistency.py --write-public-artifact-inventory

The second command is required because the artifact is inventoried in public-artifacts.json, so the shape cannot move without a reviewed hash change.

Boundary. Enum names, key names, counts, and contract hashes. No values, no record content, no screen text, no paths, no tuning. Mechanism, not data — the public side of AGENTS.md 4.2.

Verified locally

ruff check openadapt_flow · ruff format --check · mypy (218 files) · scripts/check_consistency.py · scripts/check_release_consistency.py · scripts/check_bundle_phi.py · pytest tests/e2e/test_free_path_e2e.py (11 passed).

Noted, not fixed here

Adjacent halt-prose findings from the same sweep, each in another lane's files:

  • openadapt-maintenance docs/concepts/durable-runtime.md:21-23 — "When the run halts (an unhandled screen, a refuted write, an escalation) ... nothing after the checkpoint was performed." False for exactly the case it names: a refuted write was delivered, and transaction.py:362-370 says so explicitly ("a postcondition is checked after the click, so an over-halt ... is proof the write was already delivered"). Separate PR.
  • openadapt-desktop docs/DECISION_PORTAL.md:134-141 describes a reject action the engine never offers (attended.py:168 has four). feat(attended): let an operator reject a halt and end the run #295 is implementing it — left alone.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM

abrichr added 3 commits July 27, 2026 23:05
…ly reports

`docs/EXECUTION_PROFILES.md` told readers that the tutorial's
`?fault=optimistic` probe terminates `HALTED` / `HALTED_BEFORE_EFFECT`
"with nothing persisted", and cited `tests/e2e/test_free_path_e2e.py`
as the pin. That test pins `RECONCILIATION_REQUIRED`, and has since #280.

The prose is what is wrong; the code is right, and deliberately so. #281
wrote the paragraph, #280 then made `HALTED_BEFORE_EFFECT` require
positively established absence for every declared effect of every
consequential step, flipped the test, and rewrote the taxonomy table in
this same file -- but not this paragraph. The document has been
self-contradictory since: line 77 claimed `HALTED_BEFORE_EFFECT` for a
step that reached actuation, while line 99 said such a step "does not
qualify".

Measured on the real probe: the save declares 2 mined effects, the
retained verifier returns evidence for 1 (`refuted`), and the store is in
fact empty -- but only the test's out-of-band `GET /api/db` snapshot
knows that. The runtime never held absence evidence for the second
effect, so it reports `RECONCILIATION_REQUIRED`.

This matters more than a stale line. `HALTED_BEFORE_EFFECT` tells an
operator there is nothing to reconcile. Documenting it for a run that
cannot support the claim is exactly the failure the transaction taxonomy
exists to prevent, described in the file that defines the taxonomy.

Rewrites the paragraph to state both probes' real outcome, why the
optimistic case cannot claim absence, and that this is the taxonomy
working rather than a gap in it.
`VERIFIED` is one bit, and one bit is a weak release gate. A run can end
`VERIFIED` for the wrong reason and every existing assertion in
`tests/e2e/test_free_path_e2e.py` stays green:

- the structural rung stops resolving and the ladder falls to OCR or
  geometry -- weaker evidence, same verdict;
- an actuation path leaves the DOM for synthesised pointer events;
- the save loses its `irreversible` classification, and with it the armed
  identity gate;
- a heal fires on an undrifted screen;
- the mined effect contract changes shape, so `VERIFIED` is asserted
  against a different promise than the one that was reviewed.

Adds `tests/golden/tutorial_run.json`: the whole free path projected onto
a closed-vocabulary shape -- per step the resolution rung, actuation,
delivery status, identity status/mode, risk class, postcondition verdict,
heal flag and effect evidence; plus step count, `rung_counts`, identity
armed/applicable counts, required/passed contract counts, the mined
effect-contract hashes, model calls, cost, and the outcome fields. The
comparison walks both documents leaf by leaf and names the exact field
that moved rather than dumping two blobs.

Determinism: every pinned field was byte-identical across 8 local runs
under widely varying machine load. `TutorialResult.bundle_digest` is the
one unstable field (the bundle carries retained frames) and is
deliberately excluded. Nothing here is a timing, a path, or a temp dir.

Mutation proof. Drop the `field_equals` effect from
`compiler/effect_mining.py` -- the partial-save catch -- and the free path
still reaches `VERIFIED`, `test_free_path_terminates_verified_with_-
independent_effect_evidence` still passes, and this is what fails:

    changed effects_confirmed: golden=2 now=1
    changed required_contracts.effect: golden=2 now=1
    the run no longer emits steps[5].effect_evidence[1].verdict
    the run no longer emits effect_journal[0].intended_effect_contract_hashes[1]

Cost. Zero additional runner minutes. The assertion reads the report the
module-scoped `free_path` fixture already produces, so it adds a
comparison, not a run. The `e2e-browser` job now runs that module as its
FIRST step and `--ignore`s it from the directory run below, so the core
loop answers in the first ~2 minutes of a ~21-minute job instead of the
last -- ordering, not spend. No new workflow, no new job, no new required
context, and no change to any trigger.

Regenerate deliberately, never as a reflex:

    OPENADAPT_UPDATE_GOLDEN=1 pytest tests/e2e/test_free_path_e2e.py
    python scripts/check_release_consistency.py --write-public-artifact-inventory

Boundary: the artifact is enum names, key names, counts, and contract
hashes. No values, no record content, no screen text, no paths, no
tuning. Mechanism, not data -- the public side of AGENTS.md 4.2.
The adapter-result table already qualifies the `confirmed` row ("run-level
`VERIFIED` is decided by the run classifier") but not the `refuted` row,
so the table reads as though one refuted effect settles the run outcome.

It does not, and has not since #280: run-level `HALTED_BEFORE_EFFECT`
requires absence for every declared effect of every consequential step. The
tutorial's own `?fault=optimistic` probe is exactly this case -- one
refuted effect beside one unverified effect -- and terminates
`RECONCILIATION_REQUIRED`.

Adds the symmetric caveat.
@abrichr
abrichr enabled auto-merge (squash) July 28, 2026 03:06
@abrichr
abrichr force-pushed the agent/halt-truth-and-golden branch from ec4de8d to 55b1b73 Compare July 28, 2026 03:06
@abrichr
abrichr merged commit 6755c61 into main Jul 28, 2026
15 checks passed
@abrichr
abrichr deleted the agent/halt-truth-and-golden branch July 28, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant