Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable WinUI Controls Test Pipeline #17221

Merged
merged 29 commits into from Sep 22, 2023
Merged

Enable WinUI Controls Test Pipeline #17221

merged 29 commits into from Sep 22, 2023

Conversation

PureWeen
Copy link
Member

@PureWeen PureWeen commented Sep 5, 2023

Description of Change

Enables the Controls projects device tests for Mac Catalyst. A couple of things to note:

  • Due to limitations/bugs on the WinUI side we need to run these in chunks. Running all tests at once will make the application crash. For the time being running each category separately works fine. The code implemented for this isn't great, but it works. We will follow up with the WinUI team to see if we can find a cause and make it better. Eventually we can offload some of the (potentially crashing) tests to UI tests as well which might help.
  • We've disabled a number of tests to make this PR green for now, we need to revisit those and either adjust the test or the functionality.

How These Tests Get Executed / Good To Know

Because we run the tests for Controls per category, that also means the test results are divided as such. On top of that, the Windows app knows two variations: packaged (distributed through an MSIX typically) and unpackaged (more traditional exe + dlls). Because of this combination the flow of tests is a bit different than the rest.

Because tests are ran by category, each test should be in a category. Tests that are not in a category will be ran with each test run.

  • Keep in mind that the windows.cake script is used for all projects so also Core, Graphics, etc. and only this project, Controls, needs special treatment. Therefore there is some logic added in the Cake script to differentiate between Controls and the rest. To this end the Cake script has a variable isControlsProjectTestRun that checks if the project to be tested ends with Controls.DeviceTests.csproj.
  • The Controls test runner can now be started with an additional command-line argument. The first argument, which existed, is to provide the path for the test results, the second parameter, added in the PR, specifies the loop count. The loop count is provided by the Cake script.
  • When the Controls test runner is started with a loop count of -1, it will do some reflection on the test assembly to pull out the categories that are defined. Those are written to a txt file so that the Cake script can read it and determine the right loop count for the test runner.
  • From there, the actual testing starts. For 0 to {category count} the test runner is now started and the loop count is upped for each one. Each loop, for a packaged app, has a 10 second timeout. This is arbitrary and seems good enough for now. This is because packaged apps can only be started through PowerShell and going through Cake, to cmd, to PowerShell to a deployed app it seems impossible to block the thread to wait on it. Unpackaged apps do block the thread and run sequentially. To support all this there is a dedicated ControlsHeadlessTestRunner in the Windows test runner that is only used for the Controls project when running headless.
  • For each of the loops a test results file is written based on the command-line argument from above, but the category name is added to the filename. This does mean that in the Tests tab in Azure DevOps (and in the artifacts of the build) there will be a lot of test result files: one for each category for the packaged app, one for each category for the unpackaged app. That means that is there are 40 categories we will have 80 test results files in total.
  • This is not specific to this change, but for the device tests we automatically retry on failures to prevent flaky test results. This does mean that if something fails and is retried, that you might end up with 40 categories times 2 (for (un)packaged) and then times the number of retries. That makes the results a bit hard to parse. All the more reason to not have any tests fail.
  • To determine if any test has failed in any of the test result files we loop through the TestResults-*.xml files and do an XPath query to see if there is a failed count of > 0. If there is, we throw an exception from the Cake script to fail the pipeline.
  • When a process crashes the test result file is not written. To detect that we compare the number of categories that are written to the txt file from above to the test result files that are written by the test runners. If they don't match the build is failed.

Issues Fixed

Related to #11236

@Eilon Eilon added the area/testing Unit tests, device tests label Sep 6, 2023
@PureWeen PureWeen force-pushed the winui_controls_tests branch 2 times, most recently from 82c5072 to e84fca4 Compare September 7, 2023 16:15
@samhouts samhouts added this to the .NET 8 GA milestone Sep 11, 2023
@jfversluis jfversluis force-pushed the winui_controls_tests branch 2 times, most recently from fb87bf1 to 4a5df50 Compare September 14, 2023 09:55
@jfversluis jfversluis marked this pull request as ready for review September 19, 2023 14:15
@jfversluis jfversluis requested a review from a team as a code owner September 19, 2023 14:15
@jfversluis jfversluis mentioned this pull request Sep 19, 2023
40 tasks
@PureWeen PureWeen mentioned this pull request Sep 21, 2023
@PureWeen PureWeen requested review from mattleibow and rmarinho and removed request for StephaneDelcroix and rachelkang September 21, 2023 16:13
@PureWeen
Copy link
Member Author

/rebase

@PureWeen PureWeen enabled auto-merge (squash) September 22, 2023 12:07
@PureWeen PureWeen merged commit 179fdf9 into main Sep 22, 2023
47 checks passed
@PureWeen PureWeen deleted the winui_controls_tests branch September 22, 2023 14:00
@github-actions github-actions bot locked and limited conversation to collaborators Dec 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants