Skip to content

Add SubmitTestPlan action to submit to Test plans to TestRails - #6380

Merged
suhaibmujahid merged 8 commits into
mozilla:masterfrom
jpangas:add_test_rails_action
Jul 29, 2026
Merged

Add SubmitTestPlan action to submit to Test plans to TestRails#6380
suhaibmujahid merged 8 commits into
mozilla:masterfrom
jpangas:add_test_rails_action

Conversation

@jpangas

@jpangas jpangas commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Resolves #6414

This pull request introduces support for recording and applying TestRail actions for generated test plans. It adds a new action type for submitting generated test cases to TestRail, implements the handler to make the necessary API calls, and ensures these actions are properly registered and tested. Additionally, the test plan generator agent is updated to automatically record and apply these actions, and new tests are included to verify the integration.

@suhaibmujahid suhaibmujahid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thank you! Please see my comments, and we can discuss details.

Comment thread libs/hackbot-runtime/hackbot_runtime/actions/handlers/testrail_handler.py Outdated
Comment thread libs/hackbot-runtime/hackbot_runtime/actions/handlers/testrail_handler.py Outdated
Comment thread libs/hackbot-runtime/hackbot_runtime/actions/testrail.py
Comment thread libs/hackbot-runtime/hackbot_runtime/actions/testrail.py
Comment thread libs/hackbot-runtime/hackbot_runtime/actions/testrail.py Outdated
@jpangas jpangas changed the title Add TestRails action Add SubmitTestPlan action to submit to Test plans to TestRails Jul 28, 2026
@jpangas
jpangas marked this pull request as ready for review July 28, 2026 01:16
@jpangas
jpangas requested a review from a team as a code owner July 28, 2026 01:16
Copilot AI review requested due to automatic review settings July 28, 2026 01:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds first-class TestRail submission support for generated test plans by introducing a shared TestRail client library, a new recordable action (testrail.submit_test_plan), and an apply-side handler that creates a suite/section and uploads cases. It also updates the test-plan-generator agent to automatically record the TestRail submission action after generating a validated structured result.

Changes:

  • Add a new workspace library testrail-client (httpx-based) with environment-driven settings and unit tests.
  • Add testrail.submit_test_plan as a recordable action + apply-side handler, and register/expose it via the actions tool server.
  • Update the test-plan-generator agent to record the TestRail submission action, with new hackbot-runtime tests covering both recording and applying.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
uv.lock Adds testrail-client to the uv workspace lock and hackbot-runtime deps.
libs/testrail-client/pyproject.toml Defines the new testrail-client package (deps, build config, pytest config).
libs/testrail-client/testrail_client/init.py Exposes TestRailClient and TestRailSettings as the public API.
libs/testrail-client/testrail_client/config.py Adds TestRailSettings and env-backed settings loading.
libs/testrail-client/testrail_client/client.py Implements a minimal async TestRail API client and endpoint wrappers.
libs/testrail-client/tests/test_client.py Unit tests for auth, request behavior, env loading, and wrappers.
libs/hackbot-runtime/pyproject.toml Adds testrail-client as a dependency and uv workspace source.
libs/hackbot-runtime/hackbot_runtime/actions/init.py Exports the new testrail action module.
libs/hackbot-runtime/hackbot_runtime/actions/claude_sdk.py Exposes TestRail tools via the actions MCP server.
libs/hackbot-runtime/hackbot_runtime/actions/testrail.py Adds the recordable submit_test_plan tool and recording helper.
libs/hackbot-runtime/hackbot_runtime/actions/handlers/registry.py Registers testrail.submit_test_plan to its handler.
libs/hackbot-runtime/hackbot_runtime/actions/handlers/testrail_handler.py Apply-side handler that creates suite/section and uploads cases.
libs/hackbot-runtime/tests/test_claude_sdk.py Updates tool exposure tests to include the new TestRail tool.
libs/hackbot-runtime/tests/test_testrail_action.py Tests recording validation and handler registration for the action.
libs/hackbot-runtime/tests/test_testrail_handler.py Tests apply-side behavior for suite/section/case creation and failures.
agents/test-plan-generator/hackbot_agents/test_plan_generator/main.py Records the TestRail submission action after generating a validated plan.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread agents/test-plan-generator/hackbot_agents/test_plan_generator/__main__.py Outdated
Comment thread libs/hackbot-runtime/hackbot_runtime/actions/testrail.py
Comment thread libs/testrail-client/testrail_client/client.py Outdated
Comment thread libs/testrail-client/testrail_client/config.py
Comment thread libs/testrail-client/testrail_client/client.py Outdated
Comment thread libs/testrail-client/testrail_client/config.py

@suhaibmujahid suhaibmujahid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

@suhaibmujahid
suhaibmujahid enabled auto-merge (squash) July 29, 2026 18:57
@suhaibmujahid
suhaibmujahid merged commit ac6f837 into mozilla:master Jul 29, 2026
7 checks passed
@jpangas
jpangas deleted the add_test_rails_action branch July 29, 2026 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate the test-plan-generator agent with the QA workflow on TestRail

3 participants