Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests marked with OuterLoop attribute ignore -notrait category=failing #14779

Closed
mkhamoyan opened this issue May 21, 2024 · 2 comments
Closed

Comments

@mkhamoyan
Copy link
Member

While running runtime-libraries-mono outerloop tests we noticed that test cases marked with the OuterLoop attribute, which also have attributes like ActiveIssue or SkipOnPlatform, are still being executed and causing failures in the pipeline.

For example, the following test will still run on the browser platform:

[OuterLoop]
[SkipOnPlatform(TestPlatforms.Browser, "Not supported on browser")]

The command passed to Xharness contains:

-trait category=OuterLoop -notrait category=IgnoreForCI -notrait category=failing

This command should work correctly by excluding tests that shouldn't run on specific platforms.

Side Note: If we include the TestPlatforms parameter with the OuterLoop attribute, it respects the platform exclusion, and the tests are skipped as expected. For example:

[OuterLoop("Resource-intensive due to parallel processing", ~TestPlatforms.Browser)]
[SkipOnPlatform(TestPlatforms.Browser, "Not supported on browser")]
@akoeplinger
Copy link
Member

This is an xharness-only issue, tracked at dotnet/xharness#908

@mkhamoyan
Copy link
Member Author

Thanks, I'll close this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants