[WIP] The InProcDataCollector's TestCaseStart/TestCaseStop methods are not always called.

[Copilot]

Thanks for assigning this issue to me. I'm starting to work on it and will keep this PR's description up to date as I form a plan and make progress.

Original issue description:

Description

I've implemented example InProcDataCollector in order to track the test-case execution but its methods: TestCaseStart/TestCaseStop are not called for some data-driven tests.
Below is a scenario where 8 tests are executed but the InProcDataCollector is 'informed' only about 6 of them.

Steps to reproduce

• Remarks:

  • Assumption: DotNet SDK is installed on the system.
  • The attached SimpleDataCollector project is based on VsTest asset: https://github.com/microsoft/vstest/tree/main/test/TestAssets/SimpleDataCollector

• What steps can reproduce the defect?

  1. Build the attached project using SimpleDataCollector.sln
  2. Execute RunTests.cmd and observe the reported output
     (the batch will generate proper .runsettings files in the output folders (for .NetFw and .Net8) in order to use SimpleDataCollector)
     InProcDataCollector-Case.zip

Expected behavior

Expecting 8 tests to be executed and SimpleDataCollector's TestCaseStart/TestCaseStop methods are triggered for each one, so 8 times.
Traces expected to be reported by SimpleDataCollector are:

    TestSessionStart : <Configuration><Port>4312</Port></Configuration>
    TestCaseStart : TestRowsWithString_1
    TestCaseEnd : TestRowsWithString_1
    TestCaseStart : TestRowsWithString_2
    TestCaseEnd : TestRowsWithString_2
    TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
    TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
    TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
    TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
    TestCaseStart : TestRowsWithMixData_DataRow1
    TestCaseEnd : TestRowsWithMixData_DataRow1
    TestCaseStart : TestRowsWithMixData_DataRow2
    TestCaseEnd : TestRowsWithMixData_DataRow2
    TestCaseStart : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow11) parameters
    TestCaseEnd : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow11) parameters
    TestCaseStart : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow12) parameters
    TestCaseEnd : DisplayName TestWithDynamicMixDataArgs with (DynamiMixDataRow12) parameters
    TestSessionEnd

Actual behavior

Executed 8 tests but SimpleDataCollector's TestCaseStart/TestCaseStop methods are triggered only 6 times:

  TestSessionStart : <Configuration><Port>4312</Port></Configuration>
   TestCaseStart : TestRowsWithString_1
   TestCaseEnd : TestRowsWithString_1
   TestCaseStart : TestRowsWithString_2
   TestCaseEnd : TestRowsWithString_2
   TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
   TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow11, DynamiStringDataRow12) parameters
   TestCaseStart : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
   TestCaseEnd : DisplayName TestWithDynamicStringsArgs with (DynamiStringDataRow12, DynamiStringDataRow22) parameters
   TestCaseStart : TestRowsWithMixData
   TestCaseEnd : TestRowsWithMixData
   TestCaseStart : TestWithDynamicMixDataArgs
   TestCaseEnd : TestWithDynamicMixDataArgs
   TestSessionEnd

Environment

Windows10 Enterprise
VS2017.8 and .Net8 SDK
NuGet pacakges (see the attached projects):
    Microsoft.TestPlatform.XXX v17.8.0
    MSTest.XXX v3.3.1

Fixes #4997.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[nohwnd]

looks like your session timed out. Try again.

[nohwnd]

@copilot try one more time.

[Copilot]

@copilot try one more time.

I'm working on this issue now. Let me explore the codebase and investigate why InProcDataCollector's TestCaseStart/TestCaseStop methods are not being called for all data-driven tests.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• 7tjvsblobprodcus341.vsblob.vsassets.io
  • Triggering command: dotnet build test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj --no-restore (dns block)
• c50vsblobprodcus330.vsblob.vsassets.io
  • Triggering command: /home/REDACTED/work/vstest/vstest/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/vstest/vstest/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/vstest/vstest/artifacts/toolset/9.0.0-beta.25302.2.txt (dns block)
  • Triggering command: dotnet build test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj --no-restore (dns block)
• c78vsblobprodcus322.vsblob.vsassets.io
  • Triggering command: dotnet build test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj --no-restore (dns block)
• d0svsblobprodcus381.vsblob.vsassets.io
  • Triggering command: dotnet build test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj --no-restore (dns block)
• jd4vsblobprodcus366.vsblob.vsassets.io
  • Triggering command: dotnet build test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj --no-restore (dns block)
• l49vsblobprodcus358.vsblob.vsassets.io
  • Triggering command: dotnet build test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj --no-restore (dns block)
• uy6vsblobprodcus34.vsblob.vsassets.io
  • Triggering command: dotnet build test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj --no-restore (dns block)
• vb4vsblobprodcus33.vsblob.vsassets.io
  • Triggering command: dotnet build test/Microsoft.TestPlatform.CrossPlatEngine.UnitTests/Microsoft.TestPlatform.CrossPlatEngine.UnitTests.csproj --no-restore (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to my firewall allow list