Skip to content

Add CIConditionAttribute #5533

@Youssef1313

Description

@Youssef1313
Member

Summary

Consider adding a CIConditionAttribute (or ContinuousIntegrationConditionAttribute) that handles commonly known CI systems to allow the user to run a test only in CI, or exclude a test only in CI.

Background and Motivation

In some cases, a test could be flaky only in CI and temporarily disabled only in CI. Having this attribute out of the box makes it easier for MSTest users to handle this.

Proposed Feature

public sealed class CIConditionAttribute : ConditionBaseAttribute
{
    public CIConditionAttribute(ConditionMode mode);
}

This should handle GitHub Actions, Azure Pipelines, AppVeyor, Travis, CircleCI, Jenkins, ...

Alternative Designs

Activity

added
Area: MSTestIssues with MSTest that are not specific to more refined area (e.g. analyzers or assertions)
on Apr 28, 2025
nohwnd

nohwnd commented on Apr 28, 2025

@nohwnd
Member

We already have this detection in telemetry, it would be nice to re-use the class in some form (e.g. linking it because this is mstest specific feature).

https://github.com/dotnet/sdk/blob/main/src/Cli/dotnet/Telemetry/CIEnvironmentDetectorForTelemetry.cs

Azure DevOps Reporter extension could also use it if it added detection specific to each CI provider (it needs to know if it runs on azdo), same for the colored outputter that uses ansi, but does not render progress that will be needed for this PR #5535

The logic is taken from dotnet and documented here: https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry#continuous-integration-detection

nohwnd

nohwnd commented on Apr 28, 2025

@nohwnd
Member

How would the actual usage look like? I think we can assume that most users just use single CI and they don't way to specify it. How would inverse usage (run just locally) look like?

nohwnd

nohwnd commented on Apr 29, 2025

@nohwnd
Member

One more usage for TF reporter is the coloring in CI with simplified ANSI, in TerminalTest reporter. Linking to this comment from there.

Youssef1313

Youssef1313 commented on Apr 29, 2025

@Youssef1313
MemberAuthor

How would the actual usage look like? I think we can assume that most users just use single CI and they don't way to specify it. How would inverse usage (run just locally) look like?

This is supported by ConditionMode in ConditionBaseAttribute already (that's how you can include/exclude OS in OSConditionAttribute today)

nohwnd

nohwnd commented on Apr 30, 2025

@nohwnd
Member

OSConditionAttribute requires the OS to be specified right? For CI I would expect to not provide anything by default, because I am probably running on just 1 CI system, so which one exactly it is, is not relevant until I am trying to run on 2 or more, and have different rules for each.

Youssef1313

Youssef1313 commented on Apr 30, 2025

@Youssef1313
MemberAuthor

@nohwnd Yes we can start by not requiring to specify any specific CI system. So the attribute just requires ConditionMode.

Later on, we can add extra constructor overloads (or properties) to the attribute for specific CI systems. In most cases, one wouldn't need to special case a specific CI system. But in some cases, one may need to do so. For example, Aspire runs tests on both GitHubActions and Azure Pipelines. And here is a case where they need to special case GitHub Actions:

https://github.com/dotnet/aspire/blob/4e112fd2e97fbb19ed09a6eeb79e158e2354b1b2/tests/Aspire.TestUtilities/RequiresPlaywrightAttribute.cs#L21

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Area: MSTestIssues with MSTest that are not specific to more refined area (e.g. analyzers or assertions)

Projects

No projects

Relationships

None yet

    Participants

    @nohwnd@Evangelink@Youssef1313

    Issue actions

      Add `CIConditionAttribute` · Issue #5533 · microsoft/testfx