Fix eval timeouts for migrate-mstest-v3-to-v4 scenarios#548
Conversation
|
/evaluate |
There was a problem hiding this comment.
Pull request overview
This PR addresses CI timeouts in the migrate-mstest-v3-to-v4 eval scenario by pre-warming NuGet/package restore work before the timed agent run begins, reducing repeated cold-cache restore/build overhead across parallel sessions.
Changes:
- Adds a scenario
setup.commandsstep to rundotnet restorefor the MSTest.Sdk v4 migration fixture.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/migrate-mstest-v3-to-v4/eval.yaml | Adds a setup restore command to warm NuGet cache/obj prior to the timed evaluation. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 1/1 changed files
- Comments generated: 0
|
/evaluate |
Skill Validation Results
[1]
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps ▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
/evaluate |
Skill Validation Results
[1]
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps ▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
|
/evaluate |
There was a problem hiding this comment.
Pull request overview
This PR addresses CI timeouts in the migrate-mstest-v3-to-v4 evaluation by pre-warming the NuGet cache for the MSTest.Sdk v4 fixture before the agent session begins, reducing subsequent dotnet build/dotnet restore latency during the timed portion of the scenario.
Changes:
- Add a
setup.commandsstep to rundotnet restorefor the affected MSTest.Sdk v4 migration scenario. - Ensure dependency resolution happens before agent execution to avoid consuming scenario timeout budget.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/migrate-mstest-v3-to-v4/eval.yaml | Adds a setup-time dotnet restore to warm NuGet/obj for the MSTest.Sdk v4 scenario, mitigating CI timeouts. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 1/1 changed files
- Comments generated: 0
Skill Validation Results
[1]
Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps ▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
- Add dotnet restore setup commands to scenarios 3 (multiple breaking changes) and 5 (CallerInfo) to pre-warm NuGet cache before agent runs - Increase timeout for CallerInfo scenario (180s -> 240s) and multiple breaking changes scenario (240s -> 300s) to accommodate baseline agent iteration cycles - Add new scenario 10: Fix sealed custom TestMethodAttribute with Timeout changes, with fixture files and dotnet restore pre-warming
Evangelink
force-pushed
the
fix/mstest-sdk-eval-timeout
branch
from
f7fc9de to
79e9e62
Compare
Evangelink
changed the title
There was a problem hiding this comment.
Pull request overview
This PR updates the migrate-mstest-v3-to-v4 eval to avoid CI timeouts caused by slow first-time NuGet restores (notably for MSTest.Sdk-based projects) by pre-warming dependencies during scenario setup.
Changes:
- Add
setup.commands: ["dotnet restore"]to multiple scenarios so package restore happens before the agent session timeout window. - Increase timeouts for two existing scenarios to better accommodate slower CI environments.
- Add a new Goal 10 eval scenario (plus fixture files) covering a sealed
TestMethodAttributeoverride andTestTimeout/Timeoutbreaking changes.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/migrate-mstest-v3-to-v4/fixtures/v3-sealed-testmethod/TestProject.csproj | New fixture project referencing MSTest 4.1.0 to support the added sealed-attribute scenario. |
| tests/dotnet-test/migrate-mstest-v3-to-v4/fixtures/v3-sealed-testmethod/PerformanceTests.cs | New fixture demonstrating Execute override + [Timeout(TestTimeout.Infinite)] usage for migration guidance. |
| tests/dotnet-test/migrate-mstest-v3-to-v4/eval.yaml | Adds pre-restore setup commands, adjusts a couple timeouts, and introduces a new Goal 10 scenario. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 3/3 changed files
- Comments generated: 0
Resolve merge conflicts in eval.yaml: - Keep detailed rubric and timeout:300 for Goal 3 (multiple breaking changes) - Include main's new Goal 10 (verified package update with build and test) - Use v3-sealed-testmethod fixture with dotnet restore for sealed scenario - Include main's new Goals 12-15 (display name, IsInstanceOfType, behavioral changes, v3 recognition)
The plugin run timed out at 240s during local evaluation. Increase to 300s to match other complex multi-fix scenarios.
There was a problem hiding this comment.
Pull request overview
Updates the migrate-mstest-v3-to-v4 evaluation suite to reduce CI timeouts by pre-warming NuGet restores, increasing scenario timeouts for longer baseline runs, and adding a new sealed TestMethodAttribute migration scenario.
Changes:
- Add
setup.commands: dotnet restoreto several scenarios to reduce cold-cache NuGet latency. - Increase timeouts for the most complex scenarios to provide more headroom on slower CI runners.
- Add a new “sealed custom
TestMethodAttribute+Timeout(TestTimeout.Infinite)” fixture and evaluation scenario.
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/migrate-mstest-v3-to-v4/eval.yaml | Adds restore pre-warming, adjusts timeouts, and introduces a new sealed-attribute migration scenario. |
| tests/dotnet-test/migrate-mstest-v3-to-v4/fixtures/v3-sealed-testmethod/TestProject.csproj | New fixture project used by the sealed TestMethodAttribute scenario. |
| tests/dotnet-test/migrate-mstest-v3-to-v4/fixtures/v3-sealed-testmethod/PerformanceTests.cs | New fixture code intentionally using v3-era APIs (Execute, TestTimeout.Infinite) to drive migration guidance. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 3/3 changed files
- Comments generated: 4
|
/evaluate |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps ▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Problem
Several
migrate-mstest-v3-to-v4evaluation scenarios were timing out on CI:Root Cause
Two contributing factors:
dotnet buildwhich triggers NuGet package downloads. MSTest packages + MSTest.Sdk resolution adds significant latency on CI runners with cold caches.Fix
Pre-warm NuGet cache
Added
dotnet restoresetup commands to scenarios 3 (multiple breaking changes), 5 (CallerInfo), and the new scenario 10 (sealed TestMethod). This runs before the agent starts, so NuGet packages are already cached when the agent runsdotnet build.Increase timeouts for complex scenarios
New scenario
Added "Fix sealed custom TestMethodAttribute with Timeout changes" (Goal 10) with:
TimedTestMethodAttributesubclass that overridesExecute[Timeout(TestTimeout.Infinite)]usagesfixtures/v3-sealed-testmethod/dotnet restorepre-warmingVerification
All 4 scenarios pass locally with no timeouts: