Add initial CI pipeline with UI tests and axe scanning #1289

marcelwgn · 2023-05-15T19:21:01Z

This PR adds the initial CI pipeline with azure devops, re-enable the UI tests and add Axe Accessibility scanning.

kmahone · 2023-05-30T23:36:30Z

We should avoid adding a .pfx key to the repo. This can cause issues, since they expire and also get flagged by scanning tools as being 'secrets' even in cases such as this where it is just for testing.

In WinUI, we use a script to generate the .pfx key at build time GenerateTestPfx.ps1
A similar approach might work well here.

In reply to: 1569277559

WinUIGallery/Package.WAP.Dev.appxmanifest

kmahone · 2023-05-30T23:43:05Z

UITests/TestBase.cs

+			search.SendKeys(name);
+			Thread.Sleep(1_000);
+			Session.FindElementByName(name).Click();
+			Thread.Sleep(5_000);


Can we do something better than thread sleeps here? This seems unreliable.

E.g. can we wait for a certain piece of UI to update instead?

I'm not really sure how you would do that with selenium, if you have any pointers I would love to do that.

In the WinUI codebase in MUXControlsTestApp we have a WaitForIdle helper function. However, there is a significant amount that goes into that so it would be beyond the scope of this PR.
Here is the code if you are curious: https://github.com/microsoft/microsoft-ui-xaml/blob/main/test/testinfra/AppTestAutomationHelpers/IdleSynchronizer.h

In lieu of that, one other approach is to poll the UI until the change that you expect shows up.
So, if you expect that clicking a button will update a textblock, instead of just inserting a sleep you can keep polling the textblock's text property until it returns the value that you expect. There would be small waits between each check. And there would be a larger overall timeout after which the operation fails.

Here is an example of that approach:
https://github.com/microsoft/microsoft-ui-xaml/blob/main/test/testinfra/MUXTestInfra/Common/WaitHelper.cs#L33

I also thought about that API, but its quite sophisticated. IIRC, part of that infra was also moved into a package right? Or was that something else.

Regarding the polling approach, I'm not sure if there is something sensible we want to wait for. The idea was to wait for the app to navigate to the page but I don't think we can "poll" for this without introducing UIA elements.

In the case of a page navigate you could wait until a known element on the page is available in the UIA tree.

kmahone · 2023-05-30T23:43:47Z

global using OpenQA.Selenium;

I don't see the advantage of doing this. It makes more sense to include the using directives in the files that require them.

In reply to: 1569283386

Refers to: UITests/GlobalUsings.cs:1 in 07c17dd. [](commit_id = 07c17dd, deletion_comment = False)

UITests/TestBase.cs

kmahone

🕐

marcelwgn · 2023-05-31T18:38:09Z

I think I addressed most of your feedback @kmahone, the global usings file is gone. accessibility testing is an explicit test now. Regarding waiting for an element, I would appreciate some pointers on how to set this up.

kmahone · 2023-05-31T20:08:43Z

I think I addressed most of your feedback @kmahone, the global usings file is gone. accessibility testing is an explicit test now. Regarding waiting for an element, I would appreciate some pointers on how to set this up.

Thanks. I've updated the comment with a suggested approach for this.

UITests/TestBase.cs

kmahone · 2023-06-01T16:21:00Z

Looks good to me. @karkarl should take a look and approve also before merge.

marcelwgn · 2023-06-01T16:25:23Z

After/before merging, of course some Azure DevOps project also needs to be configured which is not something that I can do.

kmahone · 2023-06-01T16:40:44Z

After/before merging, of course some Azure DevOps project also needs to be configured which is not something that I can do.

Yes. Karen is going to set that up.

@Scottj1s

commit 5ab6b6f Merge: 91818c9 d7c780b Author: Karen <7976322+karkarl@users.noreply.github.com> Date: Mon Aug 28 08:57:28 2023 -0700 Merge pull request microsoft#1335 from chingucoding/dev/fix-build-errors Fix build errors and CI commit d7c780b Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Sun Aug 27 18:44:27 2023 +0200 Empty commit to trigger CI commit 7f5b455 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Sun Aug 27 18:29:37 2023 +0200 Empty commit to trigger CI commit 51efdcf Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Sun Aug 27 18:04:24 2023 +0200 Trim and parameterize publishing WinUI Gallery commit b5e7c6b Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 22:38:33 2023 +0200 Revert "Use different winappdriver port" This reverts commit 17858fa. commit 9bb3080 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 22:38:03 2023 +0200 Work around WinAppDriver/Appium issue commit 17858fa Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 21:38:45 2023 +0200 Use different winappdriver port commit 62c33a4 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 21:15:06 2023 +0200 Update axe helper to find app correctly commit 909f724 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 21:13:04 2023 +0200 Revert "Update package ID reference" This reverts commit 4c8b98f. commit 4c8b98f Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 20:55:49 2023 +0200 Update package ID reference commit bdad0b0 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 20:44:47 2023 +0200 Update azure-pipelines.yml for Azure Pipelines commit 428ed39 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 20:35:16 2023 +0200 Update WinUI Gallery project to include package file for CI commit 812900a Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Fri Aug 25 17:21:42 2023 +0200 Fix build errors commit 91818c9 Author: Karen <7976322+karkarl@users.noreply.github.com> Date: Thu Aug 24 11:52:00 2023 -0700 Update SolutionGallery variable commit 4213edd Merge: 91538f5 3e58993 Author: Scott Jones <scottj1s@gmail.com> Date: Thu Aug 24 11:40:10 2023 -0700 Merge pull request microsoft#1332 from microsoft/user/karkarl/BuildFix Build Errors and Warning Fixes commit 91538f5 Merge: 3a9999b c9f6b6a Author: Karen <7976322+karkarl@users.noreply.github.com> Date: Thu Aug 24 11:29:48 2023 -0700 Merge pull request microsoft#1289 from chingucoding/dev/create-ci Add initial CI pipeline with UI tests and axe scanning commit c9f6b6a Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Thu Jun 1 18:18:19 2023 +0200 Update page navigation logic to wait for fail if header cannot be found commit 913286a Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Thu Jun 1 13:57:23 2023 +0200 Refactor wait logic commit bbae4ab Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed May 31 20:35:02 2023 +0200 Remove obsolete code commit 661a9a3 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed May 31 19:15:11 2023 +0200 Fix compiler errors commit 71e8c9d Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed May 31 18:28:41 2023 +0200 Refactor tests commit 7822b3f Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed May 31 17:54:58 2023 +0200 Update script again commit 87d6559 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed May 31 17:54:29 2023 +0200 Fix generate test pfx script commit 89fe573 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed May 31 17:50:52 2023 +0200 Move build script commit d378e34 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed May 31 17:43:06 2023 +0200 First cr feedback commit 1d345d5 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 18:53:57 2023 +0200 Fix a11y issues (or ignore them) commit 353049b Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 18:03:34 2023 +0200 Add axe testing, add license headers commit 977ec2a Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 15:21:58 2023 +0200 Fix building tests project commit 3c8f9c2 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 15:08:04 2023 +0200 Refactor tests and fix test failures commit 23933a2 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 13:34:05 2023 +0200 Update app id, fix test failures commit 345dccb Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 13:17:48 2023 +0200 Update install script commit 9bf82cc Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 13:06:47 2023 +0200 Update install script, remove duplicate build commit 6ab1257 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 12:52:39 2023 +0200 Refactor UI tests project commit 728ba04 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 12:40:25 2023 +0200 Restructure pipeline commit 981bf7a Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 12:37:37 2023 +0200 Add certificate file commit 93e6b22 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 12:27:51 2023 +0200 Update package profiles commit 0cd8314 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 12:08:44 2023 +0200 Update build steps commit 36f2001 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 12:00:00 2023 +0200 Disable publish ready to run, formatting commit 7db5cf9 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 11:45:41 2023 +0200 Bump colorcode, use nuget installer commit 2cfa73d Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 11:41:10 2023 +0200 Update to use correct restore commit 9c7feaa Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 11:37:49 2023 +0200 Try fixing restore commit 8a600a0 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Wed Apr 26 11:23:52 2023 +0200 Add initial pipelines file commit 3e58993 Author: Karen Lai <biilai@microsoft.com> Date: Wed Aug 23 11:46:46 2023 -0700 fix warnings commit 273e2a7 Author: Karen Lai <biilai@microsoft.com> Date: Wed Aug 23 11:24:47 2023 -0700 add missing namespace commit 3a9999b Merge: 81209eb 72ddf1a Author: Karen <7976322+karkarl@users.noreply.github.com> Date: Wed Aug 16 10:46:23 2023 -0700 Merge pull request microsoft#1326 from microsoft/user/karkarl/MiscA11yFixes Miscellaneous Acessibility Fixes commit 72ddf1a Author: Karen Lai <biilai@microsoft.com> Date: Wed Aug 16 10:43:31 2023 -0700 fix formatting commit db3b00c Author: Karen Lai <biilai@microsoft.com> Date: Wed Aug 16 10:35:55 2023 -0700 fix typo commit 1332979 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 15 10:02:16 2023 -0700 revert commit cf4a9f8 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 15 09:56:15 2023 -0700 changes from comments commit f8b173a Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 15 09:53:03 2023 -0700 formatting commit 81209eb Author: Pratik Anand <pratik.preet@gmail.com> Date: Mon Aug 14 22:21:09 2023 -0700 WinUI gallery unpackaged mode working + single project config for both packaged and unpackaged (microsoft#1329) Unpackaged mode no longer crashes on launch The WinUI gallery unpacakged mode was broken and crashed on launch - no matter whether building from msbuild cmd line or VS. It was happening because at some places in the code resource was getting accessed through URI which only makes sense for packaged app and not unpackaged. To remedy this, other parts of the code had separate code under UNPACKAGED which would build a non-URI path for unpackaged app config. This UNPACKAGED constant was not defined anywhere and as a result, that alternative code was not getting accessed. Some places had missing alternative code entirely. This PR removes dependence on UNPACKAGED compile time variable entirely. This PR side steps compile time variable for a easier to maintain runtime function which uses Appmodel/GetCurrentPackageId() system api to determine whether an app is running in packaged mode or unpackaged mode during runtime. Single project configuration As a need arose for cleaning up the project files for unpackaged winui gallery to work, I used this opportunity to build upon @Scottj1s 's work in unpkg-module branch and get rid of WAP configuration, have single project configuration for building any flavor of WinUI gallery. As a result, using launch configurations, you can build WinUI gallery in either packaged or unpackaged mode. I have introduced more solution configurations to aid in that. image If you want to build a Packaged winui gallery, select Packaged launch configuration and choose between Debug or Release solution configs. If you want to build a Unpackaged winui gallery, select Unpackaged launch configuration and choose between Debug-Unpackaged or Release-Unpackaged solution configs. A mix and match config will result in error because that's how Visual Studio works. commit 3468301 Author: Karen Lai <biilai@microsoft.com> Date: Mon Aug 14 18:35:54 2023 -0700 remove esc as back nav commit 11a80f2 Author: Karen Lai <biilai@microsoft.com> Date: Mon Aug 14 18:34:10 2023 -0700 add theme changed announcment commit 0189c6f Author: Karen Lai <biilai@microsoft.com> Date: Mon Aug 14 17:57:51 2023 -0700 add automationproperties.name commit 79dc855 Author: Karen Lai <biilai@microsoft.com> Date: Mon Aug 14 17:41:18 2023 -0700 add x namespace to name commit 1ffc48c Author: Pratik Anand <pratik.preet@gmail.com> Date: Wed Aug 9 15:41:30 2023 -0700 More changes for submodule (microsoft#1327) hardcoded version removed. Moved winappsdk to non-preview stable version. commit 6ddf021 Author: Karen Lai <biilai@microsoft.com> Date: Wed Aug 9 11:46:24 2023 -0700 announceactionforaccessibility commit 710c173 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 19:28:46 2023 -0700 AnnounceActionForAccessibility commit 0a6db75 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 19:20:40 2023 -0700 add success message and timer commit 0b646b1 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 18:52:22 2023 -0700 add manual focus commit 8cdcea3 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 18:05:31 2023 -0700 make copy button always visible commit 6ff1a98 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 17:48:52 2023 -0700 fix dark theme for colors pages commit a19a66f Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 17:30:40 2023 -0700 replace segemented with combobox commit 8e727e5 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 15:13:19 2023 -0700 fix theme toggle commit 2512e59 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 11:42:53 2023 -0700 correct backgroundColor commit 3a25d14 Author: Karen Lai <biilai@microsoft.com> Date: Tue Aug 8 11:35:51 2023 -0700 refactor inlineColorPicker with AP.labeledBy commit f0a6a30 Author: Pratik Anand <pratik.preet@gmail.com> Date: Mon Aug 7 15:22:13 2023 -0700 Merges unpkg-submodule branch into main, make the repo submodule ready (microsoft#1325) Takes changes made by Scott Jones (@Scottj1s) to make WinUIgallery act as a submodule. It also gets rid of WAP project for a single project solution. More work needed in that area for supporting unpackaged apps. commit 9b0d8f6 Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Thu Jul 13 00:33:42 2023 +0200 Fix a11y issue on InfoBadge sample (microsoft#1310) * Add tooltip to fix a11y issue * Fix sample code * Fix faulty formatting and icon commit f0825ac Author: Marcel Wagner <marcel.alex.wagner@outlook.com> Date: Sun Jul 9 17:31:04 2023 +0200 Update URL on WebView 2 sample (microsoft#1314) commit 2b68741 Author: James Pack <jamespack@users.noreply.github.com> Date: Sun Jul 2 13:12:05 2023 -0400 Small spelling fix (microsoft#1307) commit daff1e6 Merge: a3bc984 67ecd28 Author: Niels Laute <niels.laute@live.nl> Date: Tue Jun 27 00:08:25 2023 +0200 Merge pull request microsoft#1294 from microsoft/niels9001/colortool-design-fixes [A11y] Increasing MinWidth of color textbox commit 67ecd28 Author: Niels Laute <niels.laute@live.nl> Date: Mon Jun 19 12:18:12 2023 +0200 Update AccessibilityColorContrastPage.xaml commit 181ba7f Author: Niels Laute <niels.laute@live.nl> Date: Wed Jun 7 15:55:32 2023 +0200 Fix spacing commit 4fe299c Author: Niels Laute <niels.laute@live.nl> Date: Wed May 17 10:25:04 2023 +0200 Increasing MinWidth of color textbox

marcelwgn requested a review from karkarl May 15, 2023 19:21

karkarl requested a review from kmahone May 26, 2023 23:46

kmahone reviewed May 30, 2023

View reviewed changes

WinUIGallery/Package.WAP.Dev.appxmanifest Outdated Show resolved Hide resolved

kmahone reviewed May 30, 2023

View reviewed changes

UITests/TestBase.cs Outdated Show resolved Hide resolved

kmahone reviewed May 31, 2023

View reviewed changes

UITests/TestBase.cs Show resolved Hide resolved

kmahone requested changes May 31, 2023

View reviewed changes

marcelwgn requested a review from kmahone June 1, 2023 12:24

kmahone reviewed Jun 1, 2023

View reviewed changes

UITests/TestBase.cs Show resolved Hide resolved

kmahone approved these changes Jun 1, 2023

View reviewed changes

marcelwgn mentioned this pull request Jun 9, 2023

CI: Add debug & release configured CI pipeline #1302

Closed

marcelwgn added 12 commits August 24, 2023 11:21

Add initial pipelines file

8a600a0

Try fixing restore

9c7feaa

Update to use correct restore

2cfa73d

Bump colorcode, use nuget installer

7db5cf9

Disable publish ready to run, formatting

36f2001

Update build steps

0cd8314

Update package profiles

93e6b22

Add certificate file

981bf7a

Restructure pipeline

728ba04

Refactor UI tests project

6ab1257

Update install script, remove duplicate build

9bf82cc

Update install script

345dccb

marcelwgn added 14 commits August 24, 2023 11:25

Update app id, fix test failures

23933a2

Refactor tests and fix test failures

3c8f9c2

Fix building tests project

977ec2a

Add axe testing, add license headers

353049b

Fix a11y issues (or ignore them)

1d345d5

First cr feedback

d378e34

Move build script

89fe573

Fix generate test pfx script

87d6559

Update script again

7822b3f

Refactor tests

71e8c9d

Fix compiler errors

661a9a3

Remove obsolete code

bbae4ab

Refactor wait logic

913286a

Update page navigation logic to wait for fail if header cannot be found

c9f6b6a

karkarl force-pushed the dev/create-ci branch from 5b623c7 to c9f6b6a Compare August 24, 2023 18:26

karkarl approved these changes Aug 24, 2023

View reviewed changes

karkarl merged commit 91538f5 into microsoft:main Aug 24, 2023
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial CI pipeline with UI tests and axe scanning #1289

Add initial CI pipeline with UI tests and axe scanning #1289

marcelwgn commented May 15, 2023 •

edited

Loading

kmahone commented May 30, 2023 •

edited

Loading

kmahone May 30, 2023

marcelwgn May 31, 2023

kmahone May 31, 2023

marcelwgn May 31, 2023

kmahone May 31, 2023

kmahone commented May 30, 2023 •

edited

Loading

kmahone left a comment

marcelwgn commented May 31, 2023

kmahone commented May 31, 2023

kmahone commented Jun 1, 2023

marcelwgn commented Jun 1, 2023

kmahone commented Jun 1, 2023

Add initial CI pipeline with UI tests and axe scanning #1289

Add initial CI pipeline with UI tests and axe scanning #1289

Conversation

marcelwgn commented May 15, 2023 • edited Loading

kmahone commented May 30, 2023 • edited Loading

kmahone May 30, 2023

Choose a reason for hiding this comment

marcelwgn May 31, 2023

Choose a reason for hiding this comment

kmahone May 31, 2023

Choose a reason for hiding this comment

marcelwgn May 31, 2023

Choose a reason for hiding this comment

kmahone May 31, 2023

Choose a reason for hiding this comment

kmahone commented May 30, 2023 • edited Loading

kmahone left a comment

Choose a reason for hiding this comment

marcelwgn commented May 31, 2023

kmahone commented May 31, 2023

kmahone commented Jun 1, 2023

marcelwgn commented Jun 1, 2023

kmahone commented Jun 1, 2023

marcelwgn commented May 15, 2023 •

edited

Loading

kmahone commented May 30, 2023 •

edited

Loading

kmahone commented May 30, 2023 •

edited

Loading