Migrate E2E harness from WinAppDriver to winapp ui#650
Merged
Conversation
…pp ui Replace the Appium.WebDriver + WinAppDriver.exe harness with Microsoft's winapp CLI (UIA-based, process-per-call). The app is driven via a WinAppUi subprocess wrapper; a thin Win32 SendInput InputInjector handles the input tests; a UiaPropertyReader (COM) fills winapp's property-read gaps. Base-class helper signatures are unchanged so most test bodies are untouched. - Add WinAppUi, UiElement, UiaPropertyReader, InputInjector, Keys, WinAppException. - Rewrite TestSession/WinFormsTestSession to launch the Host and capture PID+HWND; delete WinAppDriverHelper. - Re-point AppTestBase/WinFormsTestBase at WinAppUi/InputInjector. - Port all test files off OpenQA/Appium; OnTapped now uses a real pointer tap. - Remove Appium.WebDriver from the csproj and Directory.Packages.props. Input-injection tests gracefully skip (Inconclusive) when the process cannot reach the interactive input desktop (GetCursorPos/SendInput return ACCESS_DENIED for non-uiAccess processes) instead of hard-failing, mirroring the existing locked-desktop guard. They execute normally on a real interactive desktop. Native winapp verbs (winappCli #562 send-keys, #498 drag) will later replace the SendInput fallback. Result: 48 passed / 0 failed / 26 skipped, 294.2s (vs WinAppDriver baseline 66/6/2, 331.9s) -- 11.4% faster wall-clock, no failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rd opt-out Follow-up to the winapp migration, addressing review asks: - Extract IUiaPropertyReader so the CUIAutomation COM fallback can be swapped for `winapp ui get-property` once winappCli exposes the missing properties. Document the empirically-verified null-property list (winapp 0.3.2 returns null for LocalizedControlType, IsRequiredForForm, FullDescription, HeadingLevel, LandmarkType, Level, PositionInSet, SizeOfSet, ItemStatus, LiveSetting; it DOES return Name/HelpText/AccessKey) with a swap-out TODO. - Instrument WinAppUi.InvocationCount and record per-test winapp process-spawn counts (WinAppMetrics -> winapp-invocations.csv; also TestContext output) so the process-per-call overhead is measurable: ~439 spawns / 72 tests, avg 6.1. - Add E2E_SKIP_LOCK_GUARD opt-out: winapp's UIA verbs work cross-desktop, so the UIA-only tests can run even when the console session is locked/disconnected (e.g. operator on RDP); input-injection tests still self-skip via the injectability guard. Off by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ab-merge Resolve the remaining input-injection failures in the winapp-ui harness migration. All AppTests now pass (72 passed, 2 skipped, 0 failed). InputInjector: - Mark the test process Per-Monitor-V2 DPI aware at assembly load. winapp reports UIA bounds in physical pixels; a DPI-unaware process on a mixed-DPI multi-monitor desktop virtualizes GetSystemMetrics/GetCursorPos, so the SendInput absolute-coordinate normalization mapped onto the wrong pixel and injected input missed. No-op on all-100%-scale desktops (e.g. CI). - Make Foreground() robust: attach to the target thread's input queue to defeat the foreground lock, then SetForegroundWindow/BringWindowToTop/ SetActiveWindow, verify the window actually became foreground (poll ~1s), and throw a diagnosable WinAppException if it never does. A bare SetForegroundWindow silently no-ops in batch runs when focus has drifted. - Add CollapseSelectionToEnd + DragTearOffMerge helpers. DataGridTests: display-only cells fire Tapped only on the active window and the inline editor has no AutomationId, so tap cells via InputInjector (foreground + real click) and locate the editor via winapp inspect before typing through UiElement.SendKeys (foreground + UIA-focus + type). DockingInputTests: drive the tab-merge through the immediate tear-off pipeline (DragTearOffMerge) with a pre-drag-computed drop target, and select each tab before asserting since WinUI only surfaces the selected tab's content to UIA. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
These E2E tests were never executed in CI before. Add an e2e-tests job to ci.yml that installs the winapp CLI via microsoft/setup-WinAppCli, builds the test project + Host (Debug/x64 — the path TestSession.FindHostExe expects), and runs the suite. The job is gated on non-md changes like the other test jobs. winapp ui's UIA reads + the harness's robust foreground/SendInput path let these run on the interactive windows-latest hosted runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
This E2E test landed on main after the harness migration branched, so it still used the Appium API (OpenQA.Selenium, Session.FindElement/MobileBy/ WebDriverException). After merging main, port it onto the winapp harness: drop the OpenQA usings and replace the hand-rolled absence poll with the purpose-built App.WaitForGone helper. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The E2E suite (tests/Reactor.AppTests) now drives the app through the winapp CLI instead of WinAppDriver. Add a best-effort winget install of Microsoft.WinAppCli to bootstrap.ps1 (idempotent; skippable via -SkipWinAppCli) so the suite runs out of the box, plus a run hint in the closing summary. A missing winget only warns and never fails bootstrap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The E2E harness migrated to the winapp CLI; refresh the contributor-facing docs that still described the old toolchain: AGENTS.md, TESTING.md, CONTRIBUTING.md, README.md, SKILL.md, the generated testing guide (edited via its docs/_pipeline template + recompiled), and the pr-review test-coverage tier doc. Documents how to install the winapp CLI (winget Microsoft.WinAppCli or bootstrap.ps1). Historical specs/research records are left as-is. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Encapsulate the two static-field writes flagged by CodeQL behind static helpers (WinAppUi.InvocationCount via RecordInvocation; UiElement's _lastTextSendSelector via RememberTextSendAndWasRepeat), keeping the process-global semantics while moving the mutation off the instance path. - Express the genuinely clearer filters with LINQ: ContainsTypedText -> Any, FindWindowHwnd -> Where, the Edit-finder inner loop -> Where, and the DockingTearOff diagnostics Read helper -> Select + FirstOrDefault. The recursive ancestor-tab walk and the outer window loop keep their explicit foreach: they guard on out-var TryGetProperty / accumulate via ref, where LINQ would reduce clarity rather than improve it. No behavior change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the tests/Reactor.AppTests E2E harness from Appium/WinAppDriver to Microsoft’s winapp ui CLI, replacing the persistent driver session with per-call CLI invocations plus a small Win32 SendInput fallback for scenarios winapp ui can’t synthesize (typing/drag/press-hold). It also updates CI + contributor docs so the new harness is installable and runnable by default.
Changes:
- Replace Appium session plumbing with a new
WinAppUiprocess wrapper,UiElementshim, and UIA COM fallback reader for propertieswinapp uican’t surface. - Add
InputInjector(SendInput) for keystroke-level input + drag/gesture scenarios, and add per-test winapp invocation metrics logging. - Update repo docs + bootstrap + CI workflow to install and run
winapp ui-driven E2E tests.
Reviewed changes
Copilot reviewed 42 out of 42 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/Reactor.AppTests/Tests/WrapElementSlotTests.cs | Switch absent-element wait to App.WaitForGone() |
| tests/Reactor.AppTests/Tests/WinFormsInteropTests.cs | Update docs/intent to winapp ui-based driving |
| tests/Reactor.AppTests/Tests/TreeViewInteractionTests.cs | Replace WinAppDriver element ops with App.Search + UIA reader |
| tests/Reactor.AppTests/Tests/NavigationViewInteractionTests.cs | Replace WinAppDriver waits/clicks with winapp ui searches |
| tests/Reactor.AppTests/Tests/InteractiveTests.cs | Update suite-level docs to winapp ui + SendInput |
| tests/Reactor.AppTests/Tests/ImmediateAndDisabledFocusableTests.cs | Remove Appium/Selenium dependencies |
| tests/Reactor.AppTests/Tests/GestureTests.cs | Use InputInjector for drag/hold; winapp click for others |
| tests/Reactor.AppTests/Tests/EventHandlerTests.cs | Use real pointer clicks/SendInput where WinUI requires it |
| tests/Reactor.AppTests/Tests/DragDropTests.cs | Replace Actions-based drag with InputInjector.Drag() |
| tests/Reactor.AppTests/Tests/DockingTearOffE2ETests.cs | Replace Desktop-root Appium session with HWND enumeration + SendInput |
| tests/Reactor.AppTests/Tests/DockingInputTests.cs | Replace tab-merge drag + typing with InputInjector + winapp helpers |
| tests/Reactor.AppTests/Tests/DevtoolsUxTests.cs | Remove Appium dependency |
| tests/Reactor.AppTests/Tests/DataGridTests.cs | Switch to cell-tap via SendInput + editor discovery via inspect |
| tests/Reactor.AppTests/Tests/ChartAccessibilityTests.cs | Update harness docs to winapp ui |
| tests/Reactor.AppTests/Tests/AccessibilityTests.cs | Swap exception type + remove implicit-wait plumbing |
| tests/Reactor.AppTests/Tests/AccessibilityInteractionTests.cs | Update element lookup to new harness types |
| tests/Reactor.AppTests/Reactor.AppTests.csproj | Remove Appium.WebDriver package reference |
| tests/Reactor.AppTests/Infrastructure/WinFormsTestSession.cs | New WinForms host bootstrap via PID/HWND + WinAppUi |
| tests/Reactor.AppTests/Infrastructure/WinFormsTestBase.cs | Replace driver helpers with WinAppUi/UiElement equivalents |
| tests/Reactor.AppTests/Infrastructure/WinAppUi.cs | New winapp CLI wrapper (process + JSON parsing + waits/actions) |
| tests/Reactor.AppTests/Infrastructure/WinAppMetrics.cs | Best-effort CSV metrics for per-test winapp invocations |
| tests/Reactor.AppTests/Infrastructure/WinAppException.cs | New exception types replacing WebDriverException |
| tests/Reactor.AppTests/Infrastructure/WinAppDriverHelper.cs | Remove WinAppDriver bootstrap helper |
| tests/Reactor.AppTests/Infrastructure/UiElement.cs | New lightweight element handle (invoke/click/read/sendkeys/rect) |
| tests/Reactor.AppTests/Infrastructure/UiaPropertyReader.cs | UIA COM fallback reader for properties missing in winapp |
| tests/Reactor.AppTests/Infrastructure/TestSession.cs | Replace Appium session bootstrap with WinAppUi-based bootstrap |
| tests/Reactor.AppTests/Infrastructure/SessionInteractivityGuard.cs | Split “interactive UIA” vs “input-injectable” gating |
| tests/Reactor.AppTests/Infrastructure/Keys.cs | Replace Selenium Keys constants |
| tests/Reactor.AppTests/Infrastructure/IUiaPropertyReader.cs | Interface to isolate UIA fallback implementation |
| tests/Reactor.AppTests/Infrastructure/InputInjector.cs | New SendInput fallback (PMv2 DPI + foregrounding + drag/typing) |
| tests/Reactor.AppTests/Infrastructure/AppTestBase.cs | Rewrite base test helpers around WinAppUi/UiElement |
| TESTING.md | Update E2E suite docs from WinAppDriver to winapp ui |
| SKILL.md | Update quick commands to reference winapp ui E2E |
| README.md | Update repo layout blurb for E2E harness |
| docs/guide/testing.md | Update user guide testing tier descriptions |
| docs/_pipeline/templates/testing.md.dt | Update doc template (source of generated guide) |
| Directory.Packages.props | Remove Appium.WebDriver pinned version |
| CONTRIBUTING.md | Update contributor test command docs |
| bootstrap.ps1 | Add best-effort WinAppCli install + opt-out flag |
| AGENTS.md | Update agent docs for E2E driver |
| .github/workflows/ci.yml | Add winapp CLI setup + new E2E test job |
| .github/skills/pr-review/dimensions/test-coverage.md | Update skill docs to winapp ui E2E tier wording |
Comments suppressed due to low confidence (1)
tests/Reactor.AppTests/Infrastructure/SessionInteractivityGuard.cs:123
- Comment still refers to
WebDriverExceptioneven though the harness now throwsWinAppException. This can confuse readers when diagnosing failures.
// Only reclassify when we have positive evidence the desktop is
// unreachable. Active and Unknown both fall through and the original
// WebDriverException is rethrown — masking a real failure as
// Inconclusive on Unknown would lose signal in the diagnostic loop
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Preserve nullable AutomationIds for slug selectors, surface winapp list-windows failures, avoid raw key sentinels, and keep UIA fallback string formatting from treating unsupported COM values as real text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address PR review findings in the winapp ui E2E harness: split pointer clicks from UIA invokes, harden winapp resolution/fallbacks and SendInput handling, enforce exact UIA lookups, add helper coverage, and make CI fail if input-injection E2Es are skipped environmentally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ion tests The "Fail on skipped input-injection E2Es" gate ran a repo-root TRX lookup, but `dotnet test <csproj>` writes TestResults under the project dir, so the gate threw "AppTests TRX results were not produced" even when every test passed, turning the E2E job red on green runs. Pin the TRX with --results-directory TestResults so the gate finds it. Add [Retry(3)] to the 31 synthetic-input E2E tests (keyboard/pointer/drag/ gesture). Win32 SendInput is occasionally dropped before the Host window foregrounds on the unattended CI desktop; a real regression still fails every attempt. Removable once winappCli #562 (send-keys)/#498 (drag) ship native verbs. Also harden InputInjector.ClearViaKeyboard with a 15ms Ctrl-latch delay so a fast field can't observe a bare 'A' before the Ctrl key-down dequeues (the "ahello island" select-all race). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve the three outstanding github-code-quality findings on PR #650: - WinAppUi.ResolveWinAppExe: filter PATH entries with .Where(Path.IsPathFully Qualified) instead of an in-loop guard+continue, so the foreach target is explicitly filtered. - SessionInteractivityGuard: convert all seven user32/wtsapi32 P/Invokes from raw DllImport extern to source-generated LibraryImport (partial), the modern AOT/trim-friendly form. GetUserObjectInformation and WTSQuerySessionInformation are pinned to their -W exports because LibraryImport uses ExactSpelling and does not auto-suffix. Add AllowUnsafeBlocks (required by the generator, matching the sibling Reactor.AppTests.Host project). GetProcessWindowStation was flagged as unused but is live in Diagnose() (window- station name read-back); it is kept and modernized rather than deleted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates the
tests/Reactor.AppTestsE2E harness from Appium/WinAppDriver to Microsoft'swinappCLI (winapp uisubcommands).The full suite is green: 72 passed, 0 failed, 2 skipped.
Speed delta vs WinAppDriver baseline
Approach
WinAppUiprocess wrapper + JSON parsing. Each call is a separate process invocation (no persistent session);-w <hwnd>targets the Host window.InputInjector) for input-injection tests that winapp ui can't do (per-keystroke typing, drag/tear-off, pointer gestures). Upstream gaps are tracked by winappCli Prepare 0.1.0-preview.4 release #562 (send-keys), Reactor core has hard type references to Charting (and Docking) — violates control-family isolation, leaks ~7.8 KB into AOT retail #498 (mouse/touch/keyboard), docs: add NuGet package READMEs and wire PackageReadmeFile #563 (selectors).Notable fixes for the input-injection tests
Foreground(): AttachThreadInput to defeat the foreground lock, then verify the window actually became foreground (poll ~1s) and fail fast with a diagnosable exception. A bareSetForegroundWindowsilently no-ops in batch runs when focus drifted.Tappedon the active window and the inline editor has no AutomationId, so cells are tapped via foreground+real-click and the editor is located viainspectbefore typing through UIA focus.Scope
tests/Reactor.AppTestschanged.