-
Notifications
You must be signed in to change notification settings - Fork 278
Labels
Area: MSTestIssues with MSTest that are not specific to more refined area (e.g. analyzers or assertions)Issues with MSTest that are not specific to more refined area (e.g. analyzers or assertions)
Milestone
Description
Summary
Consider adding a CIConditionAttribute
(or ContinuousIntegrationConditionAttribute
) that handles commonly known CI systems to allow the user to run a test only in CI, or exclude a test only in CI.
Background and Motivation
In some cases, a test could be flaky only in CI and temporarily disabled only in CI. Having this attribute out of the box makes it easier for MSTest users to handle this.
Proposed Feature
public sealed class CIConditionAttribute : ConditionBaseAttribute
{
public CIConditionAttribute(ConditionMode mode);
}
This should handle GitHub Actions, Azure Pipelines, AppVeyor, Travis, CircleCI, Jenkins, ...
Alternative Designs
EvangelinkCopilot
Metadata
Metadata
Assignees
Labels
Area: MSTestIssues with MSTest that are not specific to more refined area (e.g. analyzers or assertions)Issues with MSTest that are not specific to more refined area (e.g. analyzers or assertions)
Type
Projects
Milestone
Relationships
Development
Select code repository
Activity
nohwnd commentedon Apr 28, 2025
We already have this detection in telemetry, it would be nice to re-use the class in some form (e.g. linking it because this is mstest specific feature).
https://github.com/dotnet/sdk/blob/main/src/Cli/dotnet/Telemetry/CIEnvironmentDetectorForTelemetry.cs
Azure DevOps Reporter extension could also use it if it added detection specific to each CI provider (it needs to know if it runs on azdo), same for the colored outputter that uses ansi, but does not render progress that will be needed for this PR #5535
The logic is taken from dotnet and documented here: https://learn.microsoft.com/en-us/dotnet/core/tools/telemetry#continuous-integration-detection
nohwnd commentedon Apr 28, 2025
How would the actual usage look like? I think we can assume that most users just use single CI and they don't way to specify it. How would inverse usage (run just locally) look like?
nohwnd commentedon Apr 29, 2025
One more usage for TF reporter is the coloring in CI with simplified ANSI, in TerminalTest reporter. Linking to this comment from there.
Youssef1313 commentedon Apr 29, 2025
This is supported by
ConditionMode
inConditionBaseAttribute
already (that's how you can include/exclude OS inOSConditionAttribute
today)nohwnd commentedon Apr 30, 2025
OSConditionAttribute requires the OS to be specified right? For CI I would expect to not provide anything by default, because I am probably running on just 1 CI system, so which one exactly it is, is not relevant until I am trying to run on 2 or more, and have different rules for each.
Youssef1313 commentedon Apr 30, 2025
@nohwnd Yes we can start by not requiring to specify any specific CI system. So the attribute just requires
ConditionMode
.Later on, we can add extra constructor overloads (or properties) to the attribute for specific CI systems. In most cases, one wouldn't need to special case a specific CI system. But in some cases, one may need to do so. For example, Aspire runs tests on both GitHubActions and Azure Pipelines. And here is a case where they need to special case GitHub Actions:
https://github.com/dotnet/aspire/blob/4e112fd2e97fbb19ed09a6eeb79e158e2354b1b2/tests/Aspire.TestUtilities/RequiresPlaywrightAttribute.cs#L21
FrameworkConditionAttribute
#6070