test: opt-in ability tools resolve for sandbox runs#2601
Merged
Conversation
Adds a regression test proving ToolPolicyResolver surfaces an opt-in ability-projected tool (the shape data-machine-code uses for workspace_write) for a Codebox sandbox run when the runtime declares it via allow_only and an allow-mode tool policy, and hides it otherwise. Data Machine has no sandbox-specific knowledge: 'sandbox' is an unknown mode string that normalizes away while the paired 'chat' mode carries tools. This locks that contract so the runtime tool-delivery path for coding agents cannot silently regress.
Contributor
Homeboy Results —
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a regression test for
ToolPolicyResolverproving that an opt-in, ability-projected tool (the exact shape data-machine-code uses forworkspace_write) resolves for a Codebox sandbox run when the runtime declares it viaallow_onlyand an allow-modetool_policy, and stays hidden otherwise.Why
While debugging why WP Codebox coding agents couldn't edit files, this test deterministically settled an architectural question: Data Machine must not (and does not) know about the sandbox.
sandboxis just an unknown mode string that normalizes away; the pairedchatmode carries tools. The resolver, opt-in gating, and projection chain all work for the sandbox argument shape — this locks that contract so the runtime tool-delivery path can't silently regress.The test projects a real registered ability (
datamachine/get-wordpress-post) under an opt-in tool name and asserts:modes: ['sandbox','chat']+allow_only+ allow-modetool_policyTesting
php tests/Unit/AI/Tools/SandboxOptInToolResolutionTest.php(viahomeboy test data-machine):OK (2 tests, 4 assertions)AI assistance