Skip to content

Conversation

@LoopedBard3
Copy link
Member

@LoopedBard3 LoopedBard3 commented Oct 3, 2025

Disable tests that consistently fail in the VM queues for performance-ci runs, but keep them for internal runs.
This should clean up the last of the consistently failing tests in our performance-ci, where these tests are failing on the Open helix queues but are successful internally.

@LoopedBard3 LoopedBard3 requested a review from Copilot October 3, 2025 21:24
@LoopedBard3 LoopedBard3 self-assigned this Oct 3, 2025
@LoopedBard3 LoopedBard3 added bug Something isn't working impact test coverage test maybe disabled due to this issue labels Oct 3, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Modifies performance test configurations to disable flaky scenarios in VM queues for performance-ci runs while maintaining them for internal runs. The changes improve CI reliability by excluding tests that consistently fail in VM environments due to resource constraints or flakiness.

  • Added conditional gating for three performance scenarios based on team project type
  • Introduced IncludeVMFlakyScenarios property as an override mechanism
  • Added explanatory comments documenting the reasoning for each exclusion

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
eng/performance/sdk_scenarios.proj Gates Inner Loop Blazor Client scenario to internal runs due to higher resource usage
eng/performance/scenarios.proj Gates Static Winforms Template and WinForms Large scenarios to internal runs due to VM flakiness

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@LoopedBard3 LoopedBard3 marked this pull request as ready for review October 3, 2025 21:25
@LoopedBard3 LoopedBard3 enabled auto-merge (squash) October 3, 2025 21:34
@DrewScoggins
Copy link
Member

Failures seem unrelated. Not sure if we want to go ahead and merge.

@xtqqczze
Copy link
Contributor

xtqqczze commented Oct 4, 2025

These tests are not consistently failing, all tests were successful in https://github.com/dotnet/performance/pull/4998/checks?check_run_id=51930408911.

@LoopedBard3
Copy link
Member Author

Failures seem unrelated. Not sure if we want to go ahead and merge.

The failures are unrelated, but I am doing a rerun to see if it is just flaky.

These tests are not consistently failing, all tests were successful in https://github.com/dotnet/performance/pull/4998/checks?check_run_id=51930408911.

While not failing on every single test run they are failing on a very large portion of PRs keeping us from being able to set auto-complete and needing to override to merge, so I think this is still a worthwhile update especially as there are still a lot of tests for coverage of these scenarios.

Copy link
Member

@DrewScoggins DrewScoggins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LoopedBard3 LoopedBard3 merged commit 36e0ada into dotnet:main Oct 6, 2025
79 of 80 checks passed
@LoopedBard3 LoopedBard3 deleted the DisableConsistentlyFailingCITests branch October 6, 2025 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working impact test coverage test maybe disabled due to this issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants