Skip to content

.NET: Python: feat(python): Add local-codeact package with AST validation#6091

Draft
eavanvalkenburg wants to merge 7 commits into
microsoft:mainfrom
eavanvalkenburg:feature-local-codeact-python
Draft

.NET: Python: feat(python): Add local-codeact package with AST validation#6091
eavanvalkenburg wants to merge 7 commits into
microsoft:mainfrom
eavanvalkenburg:feature-local-codeact-python

Conversation

@eavanvalkenburg
Copy link
Copy Markdown
Member

Add agent-framework-local-codeact alpha package with AST validation for Foundry hosted agents

Copilot AI review requested due to automatic review settings May 26, 2026 16:09
@moonbox3 moonbox3 added documentation Improvements or additions to documentation python labels May 26, 2026
@github-actions github-actions Bot changed the title feat(python): Add local-codeact package with AST validation Python: feat(python): Add local-codeact package with AST validation May 26, 2026
@moonbox3
Copy link
Copy Markdown
Contributor

moonbox3 commented May 26, 2026

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
packages/local_codeact/agent_framework_local_codeact
   _bridge.py1965472%26, 30, 37–42, 54, 63, 73–75, 82, 133, 146–147, 150–152, 154, 163, 168–171, 175, 180, 183, 190, 200–201, 211–214, 218, 222–224, 243–245, 253–254, 261–262, 276–279, 281–282, 293
   _execute_code_tool.py2233882%56, 58, 84, 86, 89, 101, 109, 114, 116, 118, 126–127, 129, 145, 148, 151–154, 156, 161–162, 166, 229, 246–248, 285, 298–301, 305, 310, 315, 395–396, 438
   _files.py1203471%23, 27, 29, 37, 45–49, 70, 87–88, 94, 97–98, 106, 110–115, 131, 138–139, 143, 146–147, 149–150, 152–153, 156–157
   _instructions.py381560%16, 35, 41–45, 47–50, 55, 110–111, 113
   _provider.py34876%81, 85, 89, 93, 97, 101, 105, 109
   _runner.py1239621%21–24, 27, 30–37, 40, 44, 48, 52–60, 64–70, 72, 74–75, 89–91, 95–96, 100–113, 117–118, 120–121, 131–133, 142–143, 147–151, 154, 156–157, 159, 164–166, 168–171, 173, 184–197, 206, 210
   _types.py220100% 
   _validator.py761086%271–272, 286–287, 303–304, 308, 310, 337, 427
TOTAL37668459487% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
7403 34 💤 0 ❌ 0 🔥 1m 57s ⏱️

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new alpha Python workspace package, agent-framework-local-codeact, intended to enable CodeAct-style execution of model-generated Python in externally sandboxed environments (e.g., Foundry hosted agents), with subprocess execution, file-mount capture, and AST-based validation.

Changes:

  • Registers agent-framework-local-codeact in the Python workspace (uv/pyproject) and marks it as alpha in PACKAGE_STATUS.md.
  • Introduces LocalExecuteCodeTool / LocalCodeActProvider with subprocess runner + IPC bridge, file-mount capture helpers, and dynamic instructions.
  • Adds unit tests plus usage samples and package documentation.

Reviewed changes

Copilot reviewed 20 out of 22 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
python/uv.lock Adds the new editable workspace member and lock metadata.
python/pyproject.toml Registers the new workspace package.
python/PACKAGE_STATUS.md Marks agent-framework-local-codeact as alpha.
python/packages/local_codeact/pyproject.toml New package definition, tooling config, and test tasks.
python/packages/local_codeact/README.md Package docs, security posture, and configuration surface.
python/packages/local_codeact/AGENTS.md Package architecture and contributor notes.
python/packages/local_codeact/LICENSE MIT license for the new package.
python/packages/local_codeact/agent_framework_local_codeact/init.py Public API exports for the package.
python/packages/local_codeact/agent_framework_local_codeact/_types.py Public types for execution mode, mounts, and limits.
python/packages/local_codeact/agent_framework_local_codeact/_validator.py AST-based code validation layer.
python/packages/local_codeact/agent_framework_local_codeact/_bridge.py Parent-side subprocess bridge + tool dispatch.
python/packages/local_codeact/agent_framework_local_codeact/_runner.py Child-process runner implementing the JSON-lines protocol.
python/packages/local_codeact/agent_framework_local_codeact/_files.py Mount normalization + symlink-safe file capture.
python/packages/local_codeact/agent_framework_local_codeact/_instructions.py Dynamic CodeAct instructions and tool descriptions.
python/packages/local_codeact/agent_framework_local_codeact/_execute_code_tool.py Main execute_code tool orchestration and output shaping.
python/packages/local_codeact/agent_framework_local_codeact/_provider.py Context provider that injects the run-scoped tool + instructions.
python/packages/local_codeact/tests/local_codeact/test_validator.py Validator allow/block behavior tests.
python/packages/local_codeact/tests/local_codeact/test_local_codeact.py Tool/provider behavior, subprocess execution, mounts, and limits tests.
python/packages/local_codeact/samples/README.md Sample index and run instructions.
python/packages/local_codeact/samples/local_execute_code.py Local usage sample for direct tool invocation.
python/packages/local_codeact/samples/foundry_hosted_agent.py Foundry hosted-agent wiring sample.
python/packages/local_codeact/agent_framework_local_codeact/py.typed Marks the package as typed.

Comment thread python/packages/local_codeact/agent_framework_local_codeact/_validator.py Outdated
Comment thread python/packages/local_codeact/samples/local_execute_code.py Outdated
Comment thread python/packages/local_codeact/samples/local_execute_code.py Outdated
Comment thread python/packages/local_codeact/samples/foundry_hosted_agent.py Outdated
Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated Code Review

Reviewers: 4 | Confidence: 88%

✓ Correctness

After thorough examination of the local-codeact package implementation, I found no correctness bugs. The code demonstrates excellent engineering practices: proper error handling with early validation, safe resource cleanup using try-finally blocks and context managers, correct subprocess management with timeout handling, secure AST validation with comprehensive allow/block lists, and proper IPC serialization with JSON-safe conversions. All test assertions correctly match the implementation behavior. The Windows environment variable handling (SYSTEMROOT, COMSPEC, PATHEXT) is intentional and necessary for subprocess creation. The validator's permissive approach to user-defined functions is documented and tested. Edge cases like subprocess death, tool call failures, timeout during execution, and symlink handling are all properly managed.

✓ Security Reliability

The local CodeAct package provides defense-in-depth controls for executing LLM-generated Python code, with AST validation, subprocess isolation, and explicit environment control. The implementation is generally sound for its stated purpose (use in external sandboxes like Foundry). However, there are three reliability concerns: (1) the AST validator allows 'open' in ALLOWED_BUILTINS while blocking it in BLOCKED_BUILTINS, creating conflicting policy; (2) subprocess environment building on Windows includes parent environment keys that could leak sensitive data; (3) the validator allows delattr/setattr which could modify object internals unsafely. The package correctly disclaims being a security sandbox and documents required external isolation.

✓ Test Coverage

The test suite provides solid coverage of core functionality (subprocess execution, tool calling, validation, file capture, environment isolation). However, several edge cases and error paths lack coverage: (1) invalid input validation for constructors (empty/invalid paths, negative limits), (2) error handling for subprocess failures (invalid Python executable, runner script errors, malformed bridge responses), (3) boundary conditions for limits (exact limit sizes, total capture limits), (4) file mount edge cases (duplicate mounts, overlapping paths, permission errors), (5) race conditions in async tool calls, and (6) error recovery paths in the bridge protocol. The existing tests are well-structured and verify the happy paths thoroughly.

✓ Design Approach

The design approach is sound for the stated goal of adding AST-validated local code execution for Foundry hosted agents. The validation correctly runs before all execution paths, the subprocess bridge properly serializes concurrent tool calls via async locks, symlink handling prevents directory traversal, and the custom allow/block list semantics are clearly documented. All test cases in the diff are consistent with the implementation.


Automated review by eavanvalkenburg's agents

eavanvalkenburg added a commit to eavanvalkenburg/agent-framework that referenced this pull request May 27, 2026
- Remove 'open', 'getattr', 'setattr', 'hasattr' from ALLOWED_BUILTINS (bypass risk)
- Add these to BLOCKED_BUILTINS with explanatory comments
- Propagate AST validation settings to create_run_tool snapshot
- Terminate subprocess before raising on error messages
- Move module docstrings to file start in samples
- Remove pointless string statements from samples
- Document allowed_builtins behavior in visit_Call

Fixes all 8 review comments in PR microsoft#6091
@eavanvalkenburg
Copy link
Copy Markdown
Member Author

Review Comments Addressed

All 8 review comments have been addressed in commit a38ea7c:

Security Fixes

  1. ✅ Removed dangerous builtins from ALLOWED_BUILTINS: open, getattr, setattr, hasattr removed as they bypass AST attribute restrictions
  2. ✅ Added to BLOCKED_BUILTINS: getattr, setattr, hasattr, delattr now explicitly blocked with explanatory comments

Implementation Fixes

  1. ✅ create_run_tool propagation: Now propagates allowed_imports, blocked_imports, allowed_builtins, blocked_builtins to run-scoped tool
  2. ✅ Subprocess leak fix: Added await self._stop_process(process) before raising on error messages
  3. ✅ allowed_builtins behavior documented: Added docstring explaining that we only enforce block-list (not allow-list) for builtins to permit user-defined functions and registered tools

Sample Fixes

  1. ✅ Module docstrings moved: Moved module docstrings to file start in both samples
  2. ✅ Pointless strings removed: Removed trailing sample output string statements that triggered B018

All checks passing locally:

  • uv run poe check-packages -P local_codeact
  • uv run poe mypy -P local_codeact
  • uv run poe test -P local_codeact ✅ (46 tests)

@eavanvalkenburg
Copy link
Copy Markdown
Member Author

Python 3.10 Compatibility Fixed

Fixed timeout test that was failing on Python 3.10 due to different TimeoutError string representation.

Commit: 4760db8

Verified on:

  • Python 3.10.15 ✅
  • Python 3.12.7 ✅

All 46 tests now pass on both versions.

@moonbox3 moonbox3 added the .NET label May 27, 2026
@github-actions github-actions Bot changed the title Python: feat(python): Add local-codeact package with AST validation .NET: Python: feat(python): Add local-codeact package with AST validation May 27, 2026
eavanvalkenburg and others added 7 commits May 27, 2026 16:06
Add agent-framework-local-codeact alpha package for running LLM-generated
Python code in Foundry hosted agents and other sandboxed environments.

Key features:
- Subprocess execution by default (isolated process)
- Optional unsafe in-process mode for debugging
- AST-based allow-list code validation
- Customizable allowed/blocked imports and builtins
- Host tool bridge with framed JSON-lines IPC
- File mount system with capture and limits
- .NET portability features (python_executable, runner_script)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove 'open', 'getattr', 'setattr', 'hasattr' from ALLOWED_BUILTINS (bypass risk)
- Add these to BLOCKED_BUILTINS with explanatory comments
- Propagate AST validation settings to create_run_tool snapshot
- Terminate subprocess before raising on error messages
- Move module docstrings to file start in samples
- Remove pointless string statements from samples
- Document allowed_builtins behavior in visit_Call

Fixes all 8 review comments in PR microsoft#6091
Python 3.10's TimeoutError has a different string representation
than 3.11+. Update test to check for 'TimeoutError' instead of
specific message content.

Verified on Python 3.10.15 and 3.12.7.
- _validator.py: visit_Call now enforces ALLOWED_BUILTINS for names that
  match real Python builtins, while still treating unknown names as
  user-defined functions/registered tools. This makes the
  allowed_builtins parameter behave as a real allow-list.
- _bridge.py / _runner.py: add explicit '# nosec' markers next to the
  existing '# noqa: S102/S404' so bandit accepts the intentional
  subprocess import and exec() calls (this package's whole purpose).
- test_validator.py: add tests for unknown-builtin rejection,
  user-defined function acceptance, and custom allow-list expansion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
On Windows a freshly-killed subprocess can briefly hold the temporary
workspace directory open. Swallow OSError from temp_dir.cleanup() so the
caller still receives the proper error Content from the run and so the
timeout test passes on Windows.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ruff SIM105 prefers contextlib.suppress over try/except/pass.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous OSError-only suppression missed the RecursionError that
Python's TemporaryDirectory cleanup can raise on Windows when a freshly
killed subprocess still holds a handle to the workspace. Pass
ignore_cleanup_errors=True (Python 3.10+) so the platform stops retrying
rmtree, and broaden the outer suppression so unexpected cleanup errors
do not mask the actual run result.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@eavanvalkenburg eavanvalkenburg force-pushed the feature-local-codeact-python branch from f708ca4 to 9c91299 Compare May 27, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation .NET python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants