Repository navigation
Sandbox containment and revoked tool authority: ARQUIA-EVAL-001 open fixture #3122
argensite
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
An execution sandbox and a current permission check answer different questions. A code snippet can obey an import allowlist while a callable it was previously allowed to use is no longer authorized for a particular resource. Preserving a tool result for audit should not restore permission to call that tool again.
The smolagents secure-execution guide describes interpreter restrictions and sandbox options. A complementary, hypothetical test for an application wrapper is: authorize one external operation; revoke that grant; then deliver its delayed receipt and reconstruct the agent state. Check the external-effect count separately from the audit record. The receipt may improve knowledge of the past without authorizing another effect. This is a proposed application test, not a reported smolagents defect.
We are sharing ARQUIA-EVAL-001 v0.1.0, a framework-independent fixture with 28 deterministic synthetic cases, five deliberately faulty controls and an interface for independently written candidates. It runs offline with Python 3.10+ and the standard library. It does not require model access, API keys, payment or real actuators.
Download and inspect the frozen release, then run from its directory:
The reference should exit 0; the deliberately faulty control should exit 1. Keep complete outputs, exit codes, environment and exact version. A separate wrapper implementation or a counterexample outside the supplied cases would be particularly useful; no smolagents adapter is claimed here.
This comes from José María Garbini's ARQUÍA research proposal on governing the decision-to-action boundary. Its engineering paper, DOI 10.5281/zenodo.21791061, motivates testing authority separately from capability. The fixture's observer and ordering are simulated; a passing result does not establish sandbox security, production safety or validation of the wider framework.
Participation instructions and all four source DOI references. Publish findings under your own control and disclose your relationship to ARQUÍA. Negative results and criticism without running code are welcome. This invitation was written by an AI assistant authorized by the author; the supplied results are author-side checks, not independent evaluation.
All reactions