feat(tutorial): make the free path reach VERIFIED and emit a local receipt - #281
Merged
Conversation
…ceipt
The free tier was a closed loop. `replay` runs the Demo profile, whose contract
sets `require_effect_contracts=False`, so `classify_execution_outcome` maps
every completed demo run to `COMPLETED_UNVERIFIED` unconditionally -- the
payoff screen a new user saw literally opened with
`# WARNING local-quickstart - COMPLETED_UNVERIFIED`. `report-run` refuses
anything that is not VERIFIED ("Nothing emitted: run is not VERIFIED"). So the
growth artifact was gated behind a state the free tier could not reach. Both
halves reproduced on origin/main before any change.
The fix is not to loosen the profile. `openadapt-flow tutorial` gives the
tutorial the evidence Standard already demands, and then runs Standard:
1. A real system of record. MockMed is served through its transactional
persistence boundary (`mockmed.fault_server`) with `?fault=ok`, so a save
reaches a backend store instead of only mutating an in-page object.
`GET /api/db` reads that store and is never called by the application, so
the screen cannot influence the verifier.
2. A real effect contract. The demonstration is recorded with a
`system_of_record_reader`, so each event retains the observed before/after
record delta, and `compiler.effect_mining` DERIVES the contract from it. A
placeholder (unbound) effect is refused, never trusted.
3. A real admission. The unmodified fail-closed `evaluate_run_gate` admits the
run under Standard against the shipped `clinical-write` policy, with a
`RestRecordVerifier` supplying INDEPENDENT_SYSTEM (tier 1) evidence --
stronger than the tier 3 Standard requires.
Evidence that now exists and did not before, per run: two mined
system-of-record effect contracts on the consequential save; two confirmed
`rest` verdicts from an out-of-band read of the store; a governed run
authorization bound to the sealed bundle; and identity verified on every
identity-applicable step. `outcome_envelope.passed_contracts.effect` moves
0/0 -> 2/2. Nothing about the Demo profile changed: it still has
`require_effect_contracts=False`, `minimum_effect_tier=None`, and still cannot
report VERIFIED (pinned in tests).
The claim is checkable rather than asserted. Running the identical certified
bundle against an injected backend fault does NOT verify: `?fault=optimistic`
(screen says Saved, server rejected) gives HALTED / HALTED_BEFORE_EFFECT with
nothing persisted, and `?fault=partial` (row lands with the note dropped) gives
HALTED / RECONCILIATION_REQUIRED, because something landed and absence may not
be claimed. Both are pinned.
The local Run Receipt (`openadapt_flow.receipt`, `report-run --receipt DIR`) is
GENERATED from a closed allow-list, never redacted from the run report.
Subtractive redaction is unwinnable -- burned-in pixels, OCR text captured
precisely because it identifies a record, free-form halt reasons -- and one
missed field is a breach. Following the `openadapt-types` 0.6.0 decision
contract, `RunReceipt` is `extra="forbid"` and permits only: outcome and
transaction class (closed enums), billable, an optional operator-TYPED label
(bounded, single line, never derived from the recording), step/heal/model-call/
effect/identity counts, the over-halt counter-metric, cost, duration, the
resolution-rung histogram, evidence classes, effect tier reached, substrate,
provenance, flow/launcher versions, the bundle and receipt digests, and an
hour-truncated timestamp. No screenshot, OCR text, typed value, parameter, URL,
hostname, coordinate, workflow name, or free text can be represented. Today's
REPORT.md carries 22 step screenshots, plaintext typed values including a
credential, OCR identity text, and parameter values; none of it can reach the
receipt. The bundle digest is included so a third party can run the same public
tutorial and compare -- verifiability is the virality.
The receipt is local-only and never uploads. The success rail is unchanged and
enforced identically in both places: only VERIFIED (or a legacy unclassified
success) emits, so a COMPLETED_UNVERIFIED or HALTED run still emits nothing.
Also fixed, found only by running the composed loop from a clean wheel:
`RestRecordVerifier` imported `requests`, which is a dev dependency, so on a
real install every verify raised inside `_fetch_records`, was read as
unreadable, and HALTed. That failed safe but made `effects.kind: rest`
unusable out of the box. It now falls back to `httpx` (already a core
dependency); `requests` is still preferred when present.
The regression test nobody had: `tests/e2e/test_free_path_e2e.py` runs the whole
free path and asserts the terminal outcome, the profile, billability, zero model
calls, the confirmed effect evidence and its tier, the emitted receipt, the
absence of every PHI carrier from the receipt bytes, that Demo still cannot
verify, that `report-run` still refuses a non-VERIFIED run, and that an injected
fault does not verify. `scripts/quickstart_lifecycle.py` now runs `tutorial` in
the clean-machine wheel journey and inspects the same evidence, so the composed
loop is gated at the packaging boundary too -- the boundary where per-repo green
CI previously coexisted with a broken user-visible loop.
`COMPLETED_UNVERIFIED` remains never a success and never billable. No safety
gate was weakened.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM
This was referenced Jul 27, 2026
abrichr
added a commit
that referenced
this pull request
Jul 27, 2026
… invariant `lint_param_leakage` screens TEXT postconditions, and `include_region_stable` screens the parameterized TYPE step whose own diff is the typed glyphs. Nothing screened a LATER step whose changed region RENDERS that parameter, so a postcondition could silently encode run-specific data as if it were a stable property of the application -- a false halt on every run whose value differs, decided by the glyphs of that run's value, and very hard to diagnose. This is the defect behind the `clean` 2/3 over-halt on `step_010 Save Encounter`, region `(0, 228, 696, 223)`, in the published Flow 1.24.0 evidence. Reproduced exactly: compiling the bundled MockMed triage demo on `main` mines that same region, and it contains `Encounter saved - <note>` plus the new `Triage - <note>` row. It is not an engine regression -- MockMed ships inside the wheel and `c416b7d` added a patient banner to the New Encounter form, which made the previously-mined top band identical across the click and pushed the search down onto the note. openadapt-evals#273 corrects the published methodology claim. What the fix does, in `_postconditions`: 1. `_param_text_in_region` screens the mined region against the run's parameter values, over the region's OCR lines JOINED (OCR splits a rendered value across lines). Detection is forward-only `coverage`, deliberately asymmetric to `_contains_excluded`: an app's own literal is often a SUBSTRING of a longer value (MockMed's `Triage` inside the note `E2E triage booking three months`) and must not disqualify a stable region. Screening with the symmetric predicate disarmed `step_006`/`step_007` for exactly that reason. 2. A contaminated region is NARROWED, not immediately dropped (`_param_free_band`): the value's row is carved out and the tallest remaining slice is kept, preferring the slice with the most readable text (the destination screen's heading discriminates; a taller slice of blank page matches any blank screen). Dropping outright would have removed the step's only screen-state gate -- `tests/e2e/test_fault_model.py`'s commit-then-timeout case stops being detected. All 27 fault-model e2e cases pass unchanged. 3. Only when no usable slice survives is the postcondition dropped -- the fail-safe direction: an unverified step costs one missing check, a contaminated one false-halts every differing run. Reported, not silent: `compile_recording` logs each drop and writes `param_hygiene.json` (step id, region, MASKED matched text -- a parameter value can be PHI), removing a stale sidecar on a clean recompile. Not fatal; dropping is the correct outcome, not a compilation error. `lint_param_leakage` gains a scope note pointing at it. Also, the same class as #281 in two more verifiers: `FhirEffectVerifier` and `GraphQLRecordVerifier` imported `requests`, a dev/`windows`-extra dependency. Verified in a clean venv holding only the wheel's declared dependencies: on `main` both report `reachable=False` -- every verify raised, was read as *unreadable*, and HALTed. Fail-safe but unusable out of the box. Both now fall back to `httpx` (already core). FHIR additionally passes TLS verification per request, which `httpx` binds at client construction instead, so the fallback binds it there and omits the per-request kwarg. `RestActuator` (`runtime/actuators/api.py`) and `MockMedCompensator` (`runtime/effects/compensation.py`) carry the same assumption and are NOT fixed here: `api.py` classifies delivery uncertainty off `requests.exceptions`, so its fallback needs its own exception mapping and safety review rather than a drive-by change. Tests: `TestParamRegionHygiene` (7 cases) and `tests/test_effect_verifier_transport.py` (7 cases). Verified against a detached `origin/main` checkout: 4 of the 7 region cases and 5 of the 7 transport cases fail there and pass here. The 5 that pass on both are deliberate no-over-suppression guards. No published benchmark number changes: this changes future compiles only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM
abrichr
added a commit
that referenced
this pull request
Jul 27, 2026
… invariant (#285) `lint_param_leakage` screens TEXT postconditions, and `include_region_stable` screens the parameterized TYPE step whose own diff is the typed glyphs. Nothing screened a LATER step whose changed region RENDERS that parameter, so a postcondition could silently encode run-specific data as if it were a stable property of the application -- a false halt on every run whose value differs, decided by the glyphs of that run's value, and very hard to diagnose. This is the defect behind the `clean` 2/3 over-halt on `step_010 Save Encounter`, region `(0, 228, 696, 223)`, in the published Flow 1.24.0 evidence. Reproduced exactly: compiling the bundled MockMed triage demo on `main` mines that same region, and it contains `Encounter saved - <note>` plus the new `Triage - <note>` row. It is not an engine regression -- MockMed ships inside the wheel and `c416b7d` added a patient banner to the New Encounter form, which made the previously-mined top band identical across the click and pushed the search down onto the note. openadapt-evals#273 corrects the published methodology claim. What the fix does, in `_postconditions`: 1. `_param_text_in_region` screens the mined region against the run's parameter values, over the region's OCR lines JOINED (OCR splits a rendered value across lines). Detection is forward-only `coverage`, deliberately asymmetric to `_contains_excluded`: an app's own literal is often a SUBSTRING of a longer value (MockMed's `Triage` inside the note `E2E triage booking three months`) and must not disqualify a stable region. Screening with the symmetric predicate disarmed `step_006`/`step_007` for exactly that reason. 2. A contaminated region is NARROWED, not immediately dropped (`_param_free_band`): the value's row is carved out and the tallest remaining slice is kept, preferring the slice with the most readable text (the destination screen's heading discriminates; a taller slice of blank page matches any blank screen). Dropping outright would have removed the step's only screen-state gate -- `tests/e2e/test_fault_model.py`'s commit-then-timeout case stops being detected. All 27 fault-model e2e cases pass unchanged. 3. Only when no usable slice survives is the postcondition dropped -- the fail-safe direction: an unverified step costs one missing check, a contaminated one false-halts every differing run. Reported, not silent: `compile_recording` logs each drop and writes `param_hygiene.json` (step id, region, MASKED matched text -- a parameter value can be PHI), removing a stale sidecar on a clean recompile. Not fatal; dropping is the correct outcome, not a compilation error. `lint_param_leakage` gains a scope note pointing at it. Also, the same class as #281 in two more verifiers: `FhirEffectVerifier` and `GraphQLRecordVerifier` imported `requests`, a dev/`windows`-extra dependency. Verified in a clean venv holding only the wheel's declared dependencies: on `main` both report `reachable=False` -- every verify raised, was read as *unreadable*, and HALTed. Fail-safe but unusable out of the box. Both now fall back to `httpx` (already core). FHIR additionally passes TLS verification per request, which `httpx` binds at client construction instead, so the fallback binds it there and omits the per-request kwarg. `RestActuator` (`runtime/actuators/api.py`) and `MockMedCompensator` (`runtime/effects/compensation.py`) carry the same assumption and are NOT fixed here: `api.py` classifies delivery uncertainty off `requests.exceptions`, so its fallback needs its own exception mapping and safety review rather than a drive-by change. Tests: `TestParamRegionHygiene` (7 cases) and `tests/test_effect_verifier_transport.py` (7 cases). Verified against a detached `origin/main` checkout: 4 of the 7 region cases and 5 of the 7 transport cases fail there and pass here. The 5 that pass on both are deliberate no-over-suppression guards. No published benchmark number changes: this changes future compiles only. Claude-Session: https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
abrichr
added a commit
that referenced
this pull request
Jul 28, 2026
…ly reports `docs/EXECUTION_PROFILES.md` told readers that the tutorial's `?fault=optimistic` probe terminates `HALTED` / `HALTED_BEFORE_EFFECT` "with nothing persisted", and cited `tests/e2e/test_free_path_e2e.py` as the pin. That test pins `RECONCILIATION_REQUIRED`, and has since #280. The prose is what is wrong; the code is right, and deliberately so. #281 wrote the paragraph, #280 then made `HALTED_BEFORE_EFFECT` require positively established absence for every declared effect of every consequential step, flipped the test, and rewrote the taxonomy table in this same file -- but not this paragraph. The document has been self-contradictory since: line 77 claimed `HALTED_BEFORE_EFFECT` for a step that reached actuation, while line 99 said such a step "does not qualify". Measured on the real probe: the save declares 2 mined effects, the retained verifier returns evidence for 1 (`refuted`), and the store is in fact empty -- but only the test's out-of-band `GET /api/db` snapshot knows that. The runtime never held absence evidence for the second effect, so it reports `RECONCILIATION_REQUIRED`. This matters more than a stale line. `HALTED_BEFORE_EFFECT` tells an operator there is nothing to reconcile. Documenting it for a run that cannot support the claim is exactly the failure the transaction taxonomy exists to prevent, described in the file that defines the taxonomy. Rewrites the paragraph to state both probes' real outcome, why the optimistic case cannot claim absence, and that this is the taxonomy working rather than a gap in it.
abrichr
added a commit
that referenced
this pull request
Jul 28, 2026
…VERIFIED (#298) * fix(docs): correct the halt outcome the tutorial's fault probe actually reports `docs/EXECUTION_PROFILES.md` told readers that the tutorial's `?fault=optimistic` probe terminates `HALTED` / `HALTED_BEFORE_EFFECT` "with nothing persisted", and cited `tests/e2e/test_free_path_e2e.py` as the pin. That test pins `RECONCILIATION_REQUIRED`, and has since #280. The prose is what is wrong; the code is right, and deliberately so. #281 wrote the paragraph, #280 then made `HALTED_BEFORE_EFFECT` require positively established absence for every declared effect of every consequential step, flipped the test, and rewrote the taxonomy table in this same file -- but not this paragraph. The document has been self-contradictory since: line 77 claimed `HALTED_BEFORE_EFFECT` for a step that reached actuation, while line 99 said such a step "does not qualify". Measured on the real probe: the save declares 2 mined effects, the retained verifier returns evidence for 1 (`refuted`), and the store is in fact empty -- but only the test's out-of-band `GET /api/db` snapshot knows that. The runtime never held absence evidence for the second effect, so it reports `RECONCILIATION_REQUIRED`. This matters more than a stale line. `HALTED_BEFORE_EFFECT` tells an operator there is nothing to reconcile. Documenting it for a run that cannot support the claim is exactly the failure the transaction taxonomy exists to prevent, described in the file that defines the taxonomy. Rewrites the paragraph to state both probes' real outcome, why the optimistic case cannot claim absence, and that this is the taxonomy working rather than a gap in it. * test(e2e): pin HOW the free path reaches VERIFIED, not only that it does `VERIFIED` is one bit, and one bit is a weak release gate. A run can end `VERIFIED` for the wrong reason and every existing assertion in `tests/e2e/test_free_path_e2e.py` stays green: - the structural rung stops resolving and the ladder falls to OCR or geometry -- weaker evidence, same verdict; - an actuation path leaves the DOM for synthesised pointer events; - the save loses its `irreversible` classification, and with it the armed identity gate; - a heal fires on an undrifted screen; - the mined effect contract changes shape, so `VERIFIED` is asserted against a different promise than the one that was reviewed. Adds `tests/golden/tutorial_run.json`: the whole free path projected onto a closed-vocabulary shape -- per step the resolution rung, actuation, delivery status, identity status/mode, risk class, postcondition verdict, heal flag and effect evidence; plus step count, `rung_counts`, identity armed/applicable counts, required/passed contract counts, the mined effect-contract hashes, model calls, cost, and the outcome fields. The comparison walks both documents leaf by leaf and names the exact field that moved rather than dumping two blobs. Determinism: every pinned field was byte-identical across 8 local runs under widely varying machine load. `TutorialResult.bundle_digest` is the one unstable field (the bundle carries retained frames) and is deliberately excluded. Nothing here is a timing, a path, or a temp dir. Mutation proof. Drop the `field_equals` effect from `compiler/effect_mining.py` -- the partial-save catch -- and the free path still reaches `VERIFIED`, `test_free_path_terminates_verified_with_- independent_effect_evidence` still passes, and this is what fails: changed effects_confirmed: golden=2 now=1 changed required_contracts.effect: golden=2 now=1 the run no longer emits steps[5].effect_evidence[1].verdict the run no longer emits effect_journal[0].intended_effect_contract_hashes[1] Cost. Zero additional runner minutes. The assertion reads the report the module-scoped `free_path` fixture already produces, so it adds a comparison, not a run. The `e2e-browser` job now runs that module as its FIRST step and `--ignore`s it from the directory run below, so the core loop answers in the first ~2 minutes of a ~21-minute job instead of the last -- ordering, not spend. No new workflow, no new job, no new required context, and no change to any trigger. Regenerate deliberately, never as a reflex: OPENADAPT_UPDATE_GOLDEN=1 pytest tests/e2e/test_free_path_e2e.py python scripts/check_release_consistency.py --write-public-artifact-inventory Boundary: the artifact is enum names, key names, counts, and contract hashes. No values, no record content, no screen text, no paths, no tuning. Mechanism, not data -- the public side of AGENTS.md 4.2. * docs(effect-kit): note that a refuted STEP is not a refuted RUN The adapter-result table already qualifies the `confirmed` row ("run-level `VERIFIED` is decided by the run classifier") but not the `refuted` row, so the table reads as though one refuted effect settles the run outcome. It does not, and has not since #280: run-level `HALTED_BEFORE_EFFECT` requires absence for every declared effect of every consequential step. The tutorial's own `?fault=optimistic` probe is exactly this case -- one refuted effect beside one unverified effect -- and terminates `RECONCILIATION_REQUIRED`. Adds the symmetric caveat.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The free tier was a closed loop.
replayruns the Demo profile, whose contractsets
require_effect_contracts=False, soclassify_execution_outcomemapsevery completed demo run to
COMPLETED_UNVERIFIEDunconditionally -- thepayoff screen a new user saw literally opened with
# WARNING local-quickstart - COMPLETED_UNVERIFIED.report-runrefusesanything that is not VERIFIED ("Nothing emitted: run is not VERIFIED"). So the
growth artifact was gated behind a state the free tier could not reach. Both
halves reproduced on origin/main before any change.
The fix is not to loosen the profile.
openadapt-flow tutorialgives thetutorial the evidence Standard already demands, and then runs Standard:
persistence boundary (
mockmed.fault_server) with?fault=ok, so a savereaches a backend store instead of only mutating an in-page object.
GET /api/dbreads that store and is never called by the application, sothe screen cannot influence the verifier.
system_of_record_reader, so each event retains the observed before/afterrecord delta, and
compiler.effect_miningDERIVES the contract from it. Aplaceholder (unbound) effect is refused, never trusted.
evaluate_run_gateadmits therun under Standard against the shipped
clinical-writepolicy, with aRestRecordVerifiersupplying INDEPENDENT_SYSTEM (tier 1) evidence --stronger than the tier 3 Standard requires.
Evidence that now exists and did not before, per run: two mined
system-of-record effect contracts on the consequential save; two confirmed
restverdicts from an out-of-band read of the store; a governed runauthorization bound to the sealed bundle; and identity verified on every
identity-applicable step.
outcome_envelope.passed_contracts.effectmoves0/0 -> 2/2. Nothing about the Demo profile changed: it still has
require_effect_contracts=False,minimum_effect_tier=None, and still cannotreport VERIFIED (pinned in tests).
The claim is checkable rather than asserted. Running the identical certified
bundle against an injected backend fault does NOT verify:
?fault=optimistic(screen says Saved, server rejected) gives HALTED / HALTED_BEFORE_EFFECT with
nothing persisted, and
?fault=partial(row lands with the note dropped) givesHALTED / RECONCILIATION_REQUIRED, because something landed and absence may not
be claimed. Both are pinned.
The local Run Receipt (
openadapt_flow.receipt,report-run --receipt DIR) isGENERATED from a closed allow-list, never redacted from the run report.
Subtractive redaction is unwinnable -- burned-in pixels, OCR text captured
precisely because it identifies a record, free-form halt reasons -- and one
missed field is a breach. Following the
openadapt-types0.6.0 decisioncontract,
RunReceiptisextra="forbid"and permits only: outcome andtransaction class (closed enums), billable, an optional operator-TYPED label
(bounded, single line, never derived from the recording), step/heal/model-call/
effect/identity counts, the over-halt counter-metric, cost, duration, the
resolution-rung histogram, evidence classes, effect tier reached, substrate,
provenance, flow/launcher versions, the bundle and receipt digests, and an
hour-truncated timestamp. No screenshot, OCR text, typed value, parameter, URL,
hostname, coordinate, workflow name, or free text can be represented. Today's
REPORT.md carries 22 step screenshots, plaintext typed values including a
credential, OCR identity text, and parameter values; none of it can reach the
receipt. The bundle digest is included so a third party can run the same public
tutorial and compare -- verifiability is the virality.
The receipt is local-only and never uploads. The success rail is unchanged and
enforced identically in both places: only VERIFIED (or a legacy unclassified
success) emits, so a COMPLETED_UNVERIFIED or HALTED run still emits nothing.
Also fixed, found only by running the composed loop from a clean wheel:
RestRecordVerifierimportedrequests, which is a dev dependency, so on areal install every verify raised inside
_fetch_records, was read asunreadable, and HALTed. That failed safe but made
effects.kind: restunusable out of the box. It now falls back to
httpx(already a coredependency);
requestsis still preferred when present.The regression test nobody had:
tests/e2e/test_free_path_e2e.pyruns the wholefree path and asserts the terminal outcome, the profile, billability, zero model
calls, the confirmed effect evidence and its tier, the emitted receipt, the
absence of every PHI carrier from the receipt bytes, that Demo still cannot
verify, that
report-runstill refuses a non-VERIFIED run, and that an injectedfault does not verify.
scripts/quickstart_lifecycle.pynow runstutorialinthe clean-machine wheel journey and inspects the same evidence, so the composed
loop is gated at the packaging boundary too -- the boundary where per-repo green
CI previously coexisted with a broken user-visible loop.
COMPLETED_UNVERIFIEDremains never a success and never billable. No safetygate was weakened.
Real terminal output, clean venv, wheel-installed
REPORT.mdline 1 is now# ✅ local-quickstart — VERIFIED.Before, on
origin/main, the same loop gaveReplay COMPLETED_UNVERIFIED,# ⚠️ local-quickstart — COMPLETED_UNVERIFIED, andNothing emitted: run is not VERIFIED.🤖 Generated with Claude Code
https://claude.ai/code/session_01NyCHrzA1psrKMFfroYbzaM