test(conformance): separate ACTION provenance, authorization, and controller reporting - #162
Conversation
test(conformance): separate ACTION provenance, authorization, and controller reporting Adds provenance_status, authorization_decision, and controller_outcome to the private ACTION conformance helper's result, so a correct denial or a correct controller rejection is never confused with a broken or absent delegation chain. Fields are set in prerequisite order, per the agreed scope: authorization is reported only once provenance verifies, and controller_outcome only once authorization allows the request. provenance_status distinguishes invalid (chain/records present but a check found a defect) from missing (the record set needed to complete the check was never supplied), per the five agreed reporting cases. Cases 1-4 map onto existing ACTION-003/005/006/007/008 cases, now asserting all three new fields. Case 5 (outside helper scope) is documented in tests/conformance/README.md but not exercised by a new case in this change. Preserves the existing classification and code fields, and every existing reason code, for compatibility.
Updates tests/conformance/README.md to document the three fields, the security order, the provenance/controller boundaries (controller_outcome remains a claimed test-helper input, not cryptographically proven execution evidence), and the five minimum reporting cases.
test(conformance): add provenance/authorization/controller reporting to ACTION helper Adds provenance_status, authorization_decision, and controller_outcome to _ActionEvidenceResult, set by _verify_action_evidence() in prerequisite order: authorization only once provenance verifies, controller_outcome only once authorization allows the request. provenance_status distinguishes invalid (chain/records present but a check failed), missing (records incomplete for the claimed chain), and not_evaluated (no chain claimed at all, ACTION-016: outside this helper's scope). Updates all 13 existing ACTION assertions and adds ACTION-016 for the fifth agreed reporting case. Preserves existing classification, code, and reason codes.
docs(conformance): document ACTION provenance/authorization/controller fields Documents provenance_status, authorization_decision, and controller_outcome on the ACTION helper's result for Issue agentrust-io#144: what each field covers, the security order, the invalid/missing/not_evaluated distinction, and that controller_outcome is a claimed test-helper input, not cryptographically proven evidence. Lists the five minimum reporting cases and updates the ACTION-001 through ACTION-013 and ACTION-016 outcome column. Closes agentrust-io#144
|
Thanks for this, @Bashirloyan. Closing it for now, and it is not about the change itself. This repository asks first-time contributors to be vouched by a maintainer before opening a pull request. That is because agent-written contributions are easy to produce and expensive to review, and we would rather talk to you first than review something neither of us can explain. To get vouched: open an issue saying what you want to change and why, in your own words. A maintainer will reply, and add you with See CONTRIBUTING.md for the detail. |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
imran-siddique
left a comment
imran-siddique
left a comment
There was a problem hiding this comment.
Approving. Ran the suite rather than reading it: 47 passed on 23930803, 13 checks green once the held runs were released.
What the three axes buy, and why the ordering is the load-bearing part. Reporting provenance_status, authorization_decision and controller_outcome separately only helps if the later two cannot be guessed at when the earlier one failed, and here they cannot. With no delegation chain every axis is not_evaluated with reason outside_helper_scope. With a record set shorter than the chain, provenance_status is missing while authorization and controller are not_evaluated. An accepted controller outcome can no longer sit beside provenance that was never established.
not_evaluated is reachable rather than decorative. It appears in thirty places and the tests reach it from both directions. That matters more than the field count: an enum value nobody can produce is documentation, and the whole point of splitting the axes is to stop "we did not check" being reported as "it was fine".
The distinction in the comments is the one to keep through any later revision. "Evidence that failed verification" against "evidence that was never supplied" is why missing and invalid are separate values, and it is the first thing a future simplification would collapse.
On process: the vouch-gate closing this was ours misfiring, not anything you did. You were assigned #144 and you implemented it; being a first-time contributor should not have cost you the PR.
3 participants
What
Adds three fields (
provenance_status,authorization_decision,controller_outcome)to the private ACTION conformance helper's result, so a correct denial or a correct
controller rejection is never confused with a broken or absent delegation chain.
Why
Issue #144: the ACTION helper currently returns a broad
classificationand an exactcode, which conflates "evidence didn't verify" with "evidence verified but the actionwas denied or rejected." This separates those into three independently reported axes,
set in prerequisite order (provenance, then authorization, then controller outcome).
Closes #144
Security impact
None. This PR only changes a private test-only helper
(
tests/conformance/test_profile_conformance.py) and its documentation(
tests/conformance/README.md). It does not modify runtime code insrc/ca2a_runtimeor
src/ca2a_verify, and does not change TEE attestation, message signing, TRACE/provenance-DAG verification logic, capability-token semantics, or trust-score inputs.
It does not add a public API.
controller_outcomecontinues to reflect a claimedtest-helper input only — it is not cryptographically proven execution evidence, and
this PR does not represent it as such.
Test plan
pytestpasses —pytest tests/conformance/ -v: 47 passed. Full suite(
pytest): 592 passed, 2 skipped.ruff checkpassesmypypasses — unchanged frommain: 69 pre-existingno-untyped-callwarningsin this file (untyped local test helpers), none introduced by this change.
DCO sign-off
Developer Certificate of Origin (https://developercertificate.org).