Add test plan agent - #6237
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a new Hackbot agent, test-plan-generator, wired into the Hackbot API registry and local docker-compose workflow, to generate a Firefox QA test plan from feature details, execute it via the Firefox DevTools MCP server, and return a structured pass/fail/unsuitable result.
Changes:
- Registers a new
test-plan-generatoragent inhackbot-apiwith a dedicated input schema and associated tests. - Introduces a new agent package under
agents/test-plan-generator/including prompts, result schema/collector MCP server, Firefox install helper, DevTools MCP wiring, Dockerfile, and compose service. - Updates workspace/dependency lock and root compose include to build/run the new agent.
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Adds the new agent package as a workspace member with its dependencies. |
services/hackbot-api/tests/test_agents.py |
Adds tests for the new input schema, env serialization, and registry entry. |
services/hackbot-api/app/schemas.py |
Adds TestPlanGeneratorInputs for the new agent’s API contract. |
services/hackbot-api/app/agents.py |
Registers the new agent in AGENT_REGISTRY and exposes it via the API. |
docker-compose.yml |
Includes the agent’s compose file for local runs. |
agents/test-plan-generator/pyproject.toml |
Declares the agent distribution and its runtime dependencies. |
agents/test-plan-generator/hackbot.toml |
Documents that this agent downloads Firefox at startup (no platform prep needed). |
agents/test-plan-generator/hackbot_agents/test_plan_generator/result.py |
Defines structured result models and an in-process MCP submit_result tool. |
agents/test-plan-generator/hackbot_agents/test_plan_generator/prompts/system.md |
Adds the system prompt governing generation/execution/reporting rules. |
agents/test-plan-generator/hackbot_agents/test_plan_generator/firefox_install.py |
Implements Nightly download/install at agent startup. |
agents/test-plan-generator/hackbot_agents/test_plan_generator/devtools_mcp.py |
Builds the stdio MCP server config for the Firefox DevTools MCP server. |
agents/test-plan-generator/hackbot_agents/test_plan_generator/config.py |
Defines the allowed DevTools MCP tool allowlist for execution. |
agents/test-plan-generator/hackbot_agents/test_plan_generator/agent.py |
Orchestrates MCP servers, prompts, and result collection for the run. |
agents/test-plan-generator/hackbot_agents/test_plan_generator/__main__.py |
Reads env inputs and runs the agent entrypoint, installing Firefox first. |
agents/test-plan-generator/hackbot_agents/test_plan_generator/__init__.py |
Declares the agent subpackage. |
agents/test-plan-generator/Dockerfile |
Builds the agent image with Python deps, Node/npm, and required Firefox libs. |
agents/test-plan-generator/compose.yml |
Adds a local compose service for running the agent with env wiring. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
The lint check is failing for a file that is untouched in this PR. When I committed on my side, everything passed. Should I commit the fix it's recommending? |
…plans_agent # Conflicts: # docker-compose.yml # services/hackbot-api/app/agents.py # services/hackbot-api/app/schemas.py # uv.lock
No, it was a bug in the linter. I fixed it and I pulled from master. I also fixed the merge conflicts to save time. It will be merged once it passes CI checks. |
This pull request introduces a new "test-plan-generator" agent that automatically generates and executes Firefox QA test plans based on feature details.