Skip to content

Make agent reload test runtime-compatible#1201

Merged
stephentoub merged 3 commits into
mainfrom
stephentoub/agent-reload-test
May 5, 2026
Merged

Make agent reload test runtime-compatible#1201
stephentoub merged 3 commits into
mainfrom
stephentoub/agent-reload-test

Conversation

@stephentoub
Copy link
Copy Markdown
Collaborator

@stephentoub stephentoub commented May 5, 2026

The agent reload E2E tests were brittle against live runtime behavior because they assumed fixed custom agent names and, in some SDKs, a specific post-reload treatment of session-configured agents. In copilot-agent-runtime, the live runtime can preserve the session-configured agent after reload, while the replay runtime may return a different post-reload set.

This updates the C#, Go, Python, and Node.js reload coverage to use unique reload agent names per run, verifies the session-configured agent is present before reload, and then validates the portable contract: session.agent.reload succeeds and returns the same agent set exposed by a subsequent session.agent.list call.

Validation:

dotnet test dotnet\test\GitHub.Copilot.SDK.Test.csproj --filter "FullyQualifiedName=GitHub.Copilot.SDK.Test.E2E.RpcAgentE2ETests.Should_Call_Agent_Reload" --no-restore --logger "console;verbosity=minimal"
Set-Location go; go test ./internal/e2e -run "TestAgentSelectionRpcE2E/should_call_agent_reload"
Set-Location python; python -m pytest e2e\test_agent_and_compact_rpc_e2e.py -k test_should_call_agent_reload
Set-Location nodejs; npm test -- test\e2e\agent_and_compact_rpc.e2e.test.ts -t "should call agent reload"

Use a unique reload agent name and assert reload results match the subsequent list result instead of depending on whether the runtime preserves session-configured agents after reload.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@stephentoub stephentoub requested a review from a team as a code owner May 5, 2026 13:31
Copilot AI review requested due to automatic review settings May 5, 2026 13:31
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the .NET E2E coverage for the session.agent.reload JSON-RPC contract to be resilient to differences between the live agent runtime and the replay/runtime harness, avoiding assumptions about fixed agent names or post-reload agent set behavior.

Changes:

  • Generate a unique custom agent name per test run to avoid collisions and brittle name assumptions.
  • Assert the session-configured agent is present (and metadata matches) before reload.
  • Validate the portable contract: session.agent.reload returns the same agent set as a subsequent session.agent.list.
Show a summary per file
File Description
dotnet/test/E2E/RpcAgentE2ETests.cs Makes the agent reload E2E test runtime-compatible by using a per-run agent name and verifying reload/list consistency.

Copilot's findings

  • Files reviewed: 1/1 changed files
  • Comments generated: 0

@github-actions

This comment has been minimized.

Apply the same runtime-compatible reload test pattern to Go and Python by using unique reload agent names and asserting reload results match subsequent agent list results.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@stephentoub
Copy link
Copy Markdown
Collaborator Author

The same issue exists in the Go and Python E2E counterparts - they should get the same treatment.

Applied this across Go and Python as well: both reload tests now use unique agent names and assert the reload response matches a subsequent agent list result.

@github-actions

This comment has been minimized.

Add the Node.js counterpart for the runtime-compatible agent reload RPC test, including a unique agent name and reload/list consistency assertion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@stephentoub
Copy link
Copy Markdown
Collaborator Author

The equivalent test is missing from the Node.js SDK.

Added the Node.js reload E2E coverage too. It now follows the same pattern: unique reload agent name, pre-reload metadata assertion, and reload/list consistency check.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 5, 2026

Cross-SDK Consistency Review ✅

This PR updates the agent reload E2E test consistently across all four SDKs (Node.js, Go, Python, .NET). The key patterns are well-aligned:

Aspect Node.js Go Python .NET
Unique name randomUUID().replaceAll("-","") time.Now().UnixNano() uuid.uuid4().hex Guid.NewGuid():N
Pre-reload assertion find() + field checks assertReloadAgent() _assert_reload_agent() AssertReloadAgent()
Post-reload summary name\x00displayName sorted name\x00displayName sorted (name, display_name) tuples sorted Separate Name + DisplayName equality checks

One minor nit: The Python test fixture uses "Reload Agent" / "An agent used to validate reload" while the other three SDKs use "Reload Test Agent" / "Used by the agent reload RPC test.". Noted inline — functionally fine, just slightly asymmetric.

Generated by SDK Consistency Review Agent for issue #1201 · ● 394.8K ·

Copy link
Copy Markdown
Contributor

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generated by SDK Consistency Review Agent for issue #1201 · ● 394.8K

Comment thread python/e2e/test_agent_and_compact_rpc_e2e.py
@stephentoub stephentoub merged commit c9c3648 into main May 5, 2026
42 of 47 checks passed
@stephentoub stephentoub deleted the stephentoub/agent-reload-test branch May 5, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants