Skip to content

Add TestCategories property to ITestDataRow for per-test-case categorization #5795

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 62 commits into
base: main
Choose a base branch
from

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jun 17, 2025

This PR adds support for applying test categories to individual test data rows in parameterized tests, addressing a long-standing user request for more granular test categorization.

Changes

Core API Changes

  • Added TestCategories property to ITestDataRow interface
  • Implemented TestCategories property in TestDataRow<T> class with [DataMember] attribute
  • Property is nullable to maintain backward compatibility

Discovery Logic Updates

  • Updated TestDataSourceHelpers.TryHandleITestDataRow() to extract test categories from test data rows
  • Added backward-compatible overload to maintain existing API
  • Modified discovery logic in AssemblyEnumerator to merge test categories from test data rows with existing method/class/assembly level categories

Usage Example

[TestMethod]
[DynamicData(nameof(GetTestData), DynamicDataSourceType.Method)]
public void MyParameterizedTest(string input, string expected)
{
    Assert.AreEqual(expected, ProcessInput(input));
}

public static IEnumerable<object[]> GetTestData()
{
    // Fast unit test case
    yield return new TestDataRow<(string, string)>(("input1", "output1"))
    {
        TestCategories = new List<string> { "Unit", "Fast" },
        DisplayName = "Fast unit test"
    };
    
    // Slow integration test case
    yield return new TestDataRow<(string, string)>(("input2", "output2"))
    {
        TestCategories = new List<string> { "Integration", "Slow" },
        DisplayName = "Integration test"
    };
}

Benefits

  • Granular filtering: Can now filter tests by categories applied to individual test cases
  • Mixed scenarios: Single test method can contain both unit tests and integration tests with appropriate categories
  • Category inheritance: Test data row categories are merged with method/class/assembly level categories
  • Backward compatibility: Existing code continues to work unchanged

Test Coverage

  • Added comprehensive unit tests for TestDataRow and TestDataSourceHelpers
  • Added integration tests to verify end-to-end functionality
  • Created sample test project demonstrating real-world usage
  • Validated category filtering works correctly with test discovery

This enables scenarios like:

# Run only fast test cases
dotnet test --filter "TestCategory=Fast"

# Run only integration test cases  
dotnet test --filter "TestCategory=Integration"

Fixes #4089.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 5rqvsblobprodcus385.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:quiet /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:


💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Copilot AI and others added 3 commits June 17, 2025 08:12
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
…ng categories

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
…feature

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
@Copilot Copilot AI changed the title [WIP] Add TestCategories to ITestDataRow Add TestCategories property to ITestDataRow for per-test-case categorization Jun 17, 2025
@Copilot Copilot AI requested a review from Evangelink June 17, 2025 08:21
Copilot finished work on behalf of Evangelink June 17, 2025 08:21
…analyzer error

Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jun 20, 2025

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 5rqvsblobprodcus385.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet build src/TestFramework/TestFramework/TestFramework.csproj --no-restore (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@Copilot Copilot AI requested a review from Youssef1313 June 20, 2025 09:19
Copilot finished work on behalf of Youssef1313 June 20, 2025 09:19
@Youssef1313
Copy link
Member

@copilot There is a merge conflict. Can you fix it?

Copy link
Contributor Author

Copilot AI commented Jun 20, 2025

@copilot There is a merge conflict. Can you fix it?

The merge conflict in PublicAPI.Unshipped.txt has been resolved. I merged both the TestCategories property entries and the new Assert methods from main branch. (0ae57ea)

Copilot finished work on behalf of Youssef1313 June 20, 2025 09:23
Copy link
Contributor Author

Copilot AI commented Jun 20, 2025

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 5rqvsblobprodcus385.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@Copilot Copilot AI requested a review from Youssef1313 June 20, 2025 14:40
Copilot finished work on behalf of Youssef1313 June 20, 2025 14:40
…atting

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
SimonCropp and others added 20 commits July 14, 2025 13:16
Co-authored-by: Youssef Victor <youssefvictor00@gmail.com>
…static constructors (#5944)

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
Co-authored-by: Youssef Victor <youssefvictor00@gmail.com>
Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
…stDataRowProject

Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>
@Copilot Copilot AI requested a review from Youssef1313 July 14, 2025 18:26
Copilot finished work on behalf of Youssef1313 July 14, 2025 18:26
# Conflicts:
#	src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt
@codecov-commenter
Copy link

Codecov Report

Attention: Patch coverage is 66.66667% with 6 lines in your changes missing coverage. Please review.

Project coverage is 72.96%. Comparing base (3f0923c) to head (7c47471).
Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
....PlatformServices/Helpers/TestDataSourceHelpers.cs 44.44% 5 Missing ⚠️
src/TestFramework/TestFramework/TestDataRow.cs 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5795      +/-   ##
==========================================
- Coverage   75.70%   72.96%   -2.75%     
==========================================
  Files         615      615              
  Lines       37648    38752    +1104     
==========================================
- Hits        28502    28275     -227     
- Misses       9146    10477    +1331     
Flag Coverage Δ
Debug 72.96% <66.66%> (-2.87%) ⬇️
integration 73.33% <91.66%> (-2.50%) ⬇️
production 72.96% <66.66%> (-2.87%) ⬇️
unit 68.89% <57.14%> (-6.94%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...r.PlatformServices/Discovery/AssemblyEnumerator.cs 95.04% <100.00%> (-3.19%) ⬇️
src/TestFramework/TestFramework/TestDataRow.cs 87.50% <50.00%> (-12.50%) ⬇️
....PlatformServices/Helpers/TestDataSourceHelpers.cs 76.72% <44.44%> (-2.02%) ⬇️

... and 96 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Error loading sessions

Retrying...

Successfully merging this pull request may close these issues.

Add TestCategories to ITestDataRow
6 participants