Repository navigation
Add a coding-agent evaluation example? #4054
Greyrat-chengC
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Summary
Propose a new self-contained example under
examples/coding_agent/thatevaluates a coding agent with reproducible pytest tasks instead of an LLM
judge.
The example contains a small AgentScope coding agent, three task repositories,
a runner that copies each repository into a disposable workspace and runs
pytest, and a workspace guard that records attempted escapes. It reports
pass@1, tool calls, token usage, wall-clock time, guard blocks, and safety
violations.
Motivation
AgentScope ships coding tools (
Bash,PowerShell,Read,Write,Edit,Glob,Grep), andLocalWorkspacealready selects PowerShell on Windows.What is missing is a compact, runnable example that puts these pieces together
and makes the result measurable:
Without a task suite and a deterministic judge, "the coding agent works" is
hard to verify or compare across models, permission modes, and operating
systems. The existing
examples/demonstrate individual features (A2A, RAG,realtime, workspace), not this end-to-end loop.
What the example does
repo/, atask.jsonprompt, and an optionaloracle/known-good patch.
python main.py --oraclevalidates the tasks without a model.python main.py --runs 1runs the agent and writesagent_results.md/jsonplus one
result.jsonper run.Glob,Read,Write,Edit, and a platform shell(
PowerShellon Windows,Bashelsewhere).TraceMiddlewarerecords tool calls;WorkspaceGuardrewrites relativepaths against the task workspace, rejects paths outside it, and restricts
shell commands to
pytestplus a small read-only set.Msg.usage; no LLM judge is used.Validation
Environment: Windows 11, Python 3.12.15,
agentscope 2.0.10,deepseek-chat, one run per task.01_bug_fix02_multi_file03_injectionpython main.py --oracle: 3/3 passpython main.py --runs 1: 3/3 pass, 7 guard blocks, 0 safety violationspython -m pytest tests/test_guard.py -q: 4 passedblack --line-length 79andflake8 --extend-ignore=E203: pass on theexample's Python files
The oracle run shows each task can be fixed and that its test suite is
satisfiable; the baseline runs show the tests fail before the patch. The agent
numbers are a single run per task and are not statistically meaningful; the
README says to use
--runs 3before drawing conclusions.Scope
examples/; no core changes.requirements.txt; no new runtime dependency for the library.usage, results, and limitations.
security sandbox; real deployments should use Docker/E2B for untrusted code.
Location question
I see two plausible homes:
agentscope/examples/coding_agent/— it demonstrates core primitives(
Toolkit,Permission,Workspace, middleware) and is comparable in sizeto
examples/agent_service/examples/web_ui.agentscope-samples/evaluation/coding_agent/— if you prefer applicationsamples to live in the samples repository.
Could you confirm which one you prefer? The example is self-contained, so I can
move it without changing the code. I have not opened a PR yet.
Duplicate search
As of 2026-11-15:
examples/contains a2a, agent_service, console, long_term_memory, pipeline,rag, realtime, tui, web_ui, workspace; no coding-agent example.
coding agent,code agent, andexample; I did not find an existing proposal for this evaluation example.The closest items are [Feature]: support code agent #1054 (a stale feature request for code-agent support)
and docs(examples): RFC — ACP Agent (stdio) example design & build plan (examples/acp/) #1973 (an ACP example RFC); neither is a deterministic coding-agent
evaluation harness.
evaluation/ace_bench, a general agentbenchmark, but no coding-specific, pytest-based evaluation harness.
All reactions