Repository navigation
Testing the scope of remembered tool approval: ARQUIA-EVAL-001 open fixture #13609
argensite
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Remembering that a tool was approved once is useful only if the approval's scope is still clear. “This tool name was allowed” and “this particular action on this resource remains authorized” are different claims. A new destination, changed arguments or a revoked grant can require a different decision even when the callable has the same name.
Haystack's human-in-the-loop documentation describes confirmation through a before-tool hook and policies including asking once or on every call. A proposed application-level test is to approve an action, withdraw the corresponding grant, then invoke the same callable again after a late result has been recorded. Assert separately that the old evidence remains available and that no fresh side effect is authorized by that evidence. Approval caching and the authoritative grant store should have explicit, independently tested semantics. This is a suggested experiment for application designers, not a Haystack vulnerability report.
We are sharing ARQUIA-EVAL-001 v0.1.0, a small standalone fixture that makes the revocation/receipt contract executable. It contains 28 deterministic synthetic cases, five intentionally defective controls and an interface for independently written candidates. There is no Haystack adapter in this release; an application integration or a new counterexample would be a separate contribution.
The frozen package needs only Python 3.10+ and the standard library. Inspect the code and then run offline:
Expect exit 0 from the reference and exit 1 from that deliberately faulty control. No model, credentials, paid service or real-world actuator is required. Keep all outputs and exit codes, environment and version, including unexpected results.
The research context is José María Garbini's ARQUÍA governance proposal. Its runtime-assurance paper, DOI 10.5281/zenodo.21791061, connects operational autonomy with evidence and consequences rather than capability alone. The open call maps all four source DOI publications and welcomes reports controlled by the evaluator.
AI-assisted invitation on behalf of the author. Please disclose your relationship to ARQUÍA; negative results and contract criticism are welcome. The fixture assumes simulated observation and ordering, and detecting an escaped effect does not prevent it. Included results are author-side checks; passing does not certify Haystack, a production application or the broader ARQUÍA framework.
All reactions