Add CIConditionAttribute to support CI-specific test execution control #5797

Copilot · 2025-06-17T08:39:13Z

This PR implements CIConditionAttribute to allow developers to conditionally run or skip tests based on whether they are executing in a Continuous Integration (CI) environment.

Summary

The new attribute addresses a common need in test suites where certain tests should behave differently in CI environments - either running only in CI or being temporarily disabled in CI (e.g., for flaky tests).

Usage

[TestMethod]
[CICondition(ConditionMode.Include)]  // Run only in CI
public void PerformanceTestOnlyInCI()
{
    // Resource-intensive test that should only run in CI infrastructure
}

[TestMethod]
[CICondition(ConditionMode.Exclude)]  // Skip in CI
public void FlakyTestTemporarilyDisabledInCI()
{
    // Temporarily disable flaky test in CI while keeping it for local development
}

Implementation Details

Follows existing patterns: Inherits from ConditionBaseAttribute like OSConditionAttribute
Comprehensive CI detection: Supports major CI systems including:
- GitHub Actions (GITHUB_ACTIONS)
- Azure Pipelines (TF_BUILD)
- AppVeyor (APPVEYOR)
- Travis CI (TRAVIS)
- CircleCI (CIRCLECI)
- Jenkins (BUILD_ID + BUILD_URL)
- TeamCity (TEAMCITY_VERSION)
- AWS CodeBuild (CODEBUILD_BUILD_ID + AWS_REGION)
- Google Cloud Build (BUILD_ID + PROJECT_ID)
- JetBrains Space (JB_SPACE_API_URL)
- Generic CI (CI)
CI detection logic: Based on the well-documented detection from .NET SDK telemetry (https://learn.microsoft.com/dotnet/core/tools/telemetry#continuous-integration-detection)
Comprehensive testing: 20+ unit tests covering all CI detection scenarios and edge cases
API consistency: Updated PublicAPI files for proper API tracking

Use Cases

Flaky test management: Temporarily disable problematic tests in CI while keeping them active for local development
Resource-intensive tests: Run performance or integration tests only in CI environments with adequate resources
CI-specific scenarios: Handle tests that depend on CI infrastructure or specific CI configurations

The attribute can be applied to both test methods and test classes, supporting standard MSTest condition attribute grouping behavior.

Fixes #5533.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Youssef1313 · 2025-06-17T08:51:56Z

src/TestFramework/TestFramework/Attributes/TestMethod/CIConditionAttribute.cs

+    public override bool ShouldRun
+    {
+        get
+        {
+            bool isCI = IsCIEnvironment();
+            return Mode == ConditionMode.Include ? isCI : !isCI;
+        }
+    }


Suggested change

public override bool ShouldRun

{

get

{

bool isCI = IsCIEnvironment();

return Mode == ConditionMode.Include ? isCI : !isCI;

}

}

public override bool ShouldRun => IsCIEnvironment();

Done! Changed to public override bool ShouldRun => IsCIEnvironment(); to follow the same pattern as OSConditionAttribute. The framework handles the Mode logic in AttributeHelpers.cs. Commit: 7764d1e

…ework pattern Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>

Copilot · 2025-06-17T09:01:10Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

5rqvsblobprodcus385.vsblob.vsassets.io
- Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)
- Triggering command: dotnet build test/UnitTests/TestFramework.UnitTests/TestFramework.UnitTests.csproj --verbosity quiet (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

Youssef1313 · 2025-06-17T14:08:36Z

src/TestFramework/TestFramework/Attributes/TestMethod/CIConditionAttribute.cs

+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT license. See LICENSE file in the project root for full license information.
+
+using System.Linq;


Done! Removed the using System.Linq; statement and replaced the LINQ All() method with a traditional foreach loop to eliminate the dependency. Commit: d3fc478

Youssef1313 · 2025-06-17T14:08:54Z

src/TestFramework/TestFramework/Attributes/TestMethod/CIConditionAttribute.cs

+
+        return false;
+    }
+}


src/TestFramework/TestFramework/Attributes/TestMethod/CIConditionAttribute.cs(119,2): error SA1518: (NETCORE_ENGINEERING_TELEMETRY=Build) File is required to end with a single newline character (https://github.com/DotNetAnalyzers/StyleCopAnalyzers/blob/master/documentation/SA1518.md)

Fixed! Added the missing newline character at the end of the file to resolve the StyleCop SA1518 error. Commit: 29c801e

Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>

…te.cs Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>

Copilot · 2025-06-17T14:17:35Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

5rqvsblobprodcus385.vsblob.vsassets.io
- Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to my firewall allow list

Initial plan for issue

b2dd3d7

Copilot AI assigned Copilot and Evangelink Jun 17, 2025

Copilot started work on behalf of Evangelink June 17, 2025 08:39 View session

Copilot AI and others added 3 commits June 17, 2025 08:46

Implement CIConditionAttribute with comprehensive CI detection

5705abf

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Complete CIConditionAttribute implementation with comprehensive testing

2b6f117

Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>

Remove temporary backup file

0cfc6a8

Copilot AI changed the title ~~[WIP] Add CIConditionAttribute~~ Add CIConditionAttribute to support CI-specific test execution control Jun 17, 2025

Copilot AI requested a review from Evangelink June 17, 2025 08:50

Copilot finished work on behalf of Evangelink June 17, 2025 08:51

Youssef1313 reviewed Jun 17, 2025

View reviewed changes

Copilot started work on behalf of Youssef1313 June 17, 2025 08:52 View session

Simplify CIConditionAttribute.ShouldRun implementation following fram…

7764d1e

…ework pattern Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>

Copilot finished work on behalf of Youssef1313 June 17, 2025 09:01

Youssef1313 reviewed Jun 17, 2025

View reviewed changes

Copilot started work on behalf of Youssef1313 June 17, 2025 14:08 View session

Youssef1313 reviewed Jun 17, 2025

View reviewed changes

Remove System.Linq dependency from CIConditionAttribute

d3fc478

Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>

Copilot finished work on behalf of Youssef1313 June 17, 2025 14:13

Copilot started work on behalf of Youssef1313 June 17, 2025 14:13 View session

Fix StyleCop SA1518: Add missing newline at end of CIConditionAttribu…

29c801e

…te.cs Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com>

Copilot finished work on behalf of Youssef1313 June 17, 2025 14:17

Update CIConditionAttributeTests.cs

7242516

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add CIConditionAttribute to support CI-specific test execution control #5797

Add CIConditionAttribute to support CI-specific test execution control #5797

Uh oh!

Copilot AI commented Jun 17, 2025 •

edited

Loading

Uh oh!

Youssef1313 Jun 17, 2025

Uh oh!

Copilot AI Jun 17, 2025

Uh oh!

Copilot AI commented Jun 17, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Youssef1313 Jun 17, 2025

Uh oh!

Copilot AI Jun 17, 2025

Uh oh!

Youssef1313 Jun 17, 2025

Uh oh!

Copilot AI Jun 17, 2025

Uh oh!

Copilot AI commented Jun 17, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Add CIConditionAttribute to support CI-specific test execution control #5797

Are you sure you want to change the base?

Add CIConditionAttribute to support CI-specific test execution control #5797

Uh oh!

Conversation

Copilot AI commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Usage

Implementation Details

Use Cases

Uh oh!

Youssef1313 Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jun 17, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Youssef1313 Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Youssef1313 Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Jun 17, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Copilot AI commented Jun 17, 2025 •

edited

Loading