Skip to content

GUI Validation Methods

Jump to bottom
laqieer edited this page Mar 23, 2026 · 4 revisions

GUI Validation Methods

Comparison of two approaches for validating the Avalonia GUI: MCP Computer Use (visual screenshot-based) vs PowerShell UI Automation (structural element tree).

Test Setup

  • Target view: Unit Editor — Eirika (Unit 01) in FE8U
  • App: FEBuilderGBA Avalonia (Debug build, .NET 9.0)
  • Screen: 3240x2160 (Windows 11, 200% DPI scaling)
  • Date: 2026-03-23

Approach 1: MCP Computer Use

The MCP Computer Use server (tools/mcp-computer-use/) provides screenshot capture, mouse click, keyboard input, and window management via JSON-RPC over stdio.

What it can validate

Aspect Result
Editor opened correctly "Unit Editor" title visible with address 0x00803D64
Correct unit selected Eirika portrait rendered (teal-haired character sprite)
Identity fields Name ID=530, Name="Eirika", Unit ID=1, Class="02 Lord", Portrait=2
Base stats LV=1, Lck=5, all other stats visible and readable
Weapon levels Sword=1 (rank E), all weapon types displayed
Visual layout Sections (Identity, Base Stats, Weapon Levels) properly separated
Portrait rendering Sprite image renders correctly with proper colors
Navigation Unit list on left with portrait thumbnails, scrollable

Coordinate mapping

Screenshots are captured at full screen resolution (3240x2160) and scaled to max 1200x800.

  • Scale factor: 2.7x
  • Click coordinates: screenshot pixel position x 2.7 = screen coordinates
  • Tip: Always focus_window before clicking — the terminal steals focus

Strengths

  • Zero setup beyond existing MCP server
  • Validates what the user actually sees
  • Catches visual bugs (wrong colors, misaligned layout, missing images)
  • Works for all 356 Avalonia views without any code changes

Limitations

  • Pixel-level comparison can be flaky with font rendering differences
  • Cannot assert exact numeric values without OCR (rely on visual inspection)
  • Slower than structural queries (screenshot + network round-trip)

Approach 2: PowerShell UI Automation

Uses System.Windows.Automation (UIAutomationClient) to inspect the Avalonia window's automation element tree.

What it found

=== WINDOW FOUND ===
Name: FEBuilderGBA - FE8U.gba
ClassName: MainWindow
BoundingRect: 405,450,3236,2119

=== TOTAL DESCENDANTS: 279 ===
=== ELEMENTS WITH AUTOMATION IDs: 40 ===
Element Type Count What's visible
Text elements 34 Status bar, expander headers only
Edit fields 1 Filter textbox only
Combo boxes 0 None
List items 0 None
Buttons with AutomationId 34 Expander headers (Characters, Items, Maps...)

What it CANNOT see

The entire Unit Editor content is invisible to UI Automation:

  • No edit fields (Name ID, Unit ID, stats — all missing)
  • No combo boxes (Class ID, Affinity dropdowns — missing)
  • No list items (unit list with portraits — missing)
  • No images (portrait rendering — impossible)

Why

  1. Avalonia's AutomationPeer support is opt-in — controls without explicit automation peers are invisible
  2. Editor views don't set AutomationProperties.AutomationId on their controls
  3. Dynamically-loaded user controls inside ContentControl areas aren't exposed to the automation tree
  4. Only the main window shell (menu, status bar, expander headers) has automation support

Strengths

  • Deterministic — no visual comparison needed
  • Can read exact text values programmatically
  • Scriptable for CI/CD pipelines (headless capable)
  • Fast queries once the tree is built

Limitations

  • Blind to all editor content in current state
  • Requires adding AutomationProperties.AutomationId to hundreds of controls across 356 views
  • Avalonia's UI Automation is less mature than WinForms

Side-by-Side Comparison

Validation Target MCP Computer Use PowerShell UI Automation
Unit Editor opened Screenshot shows title + address Cannot detect
Eirika selected Portrait visible, name readable Cannot detect (0 list items)
Name ID = 530 Visible in screenshot Cannot detect (0 edit fields)
Class = 02 Lord Visible in combo box Cannot detect (0 combo boxes)
Base Stats (LV 1, Lck 5) Visible in stat fields Cannot detect
Portrait renders correctly Teal-haired sprite visible Impossible (no visual)
Layout/spacing correct Sections properly separated Impossible
Status bar text N/A `FE8U.gba
Menu structure Visible in screenshot 34 expander headers enumerable

Recommendation

Use MCP Computer Use as the primary GUI validation method for this project.

  • Most validations are visual (portrait rendering, map display, color accuracy)
  • Works out of the box with zero code changes to Avalonia views
  • Validates the actual user experience, not just the element tree
  • Already configured in .mcp.json and ready to use

Use PowerShell UI Automation only for:

  • Status bar assertions (ROM version, file size)
  • Main menu structure verification (expander presence)
  • Future: if AutomationProperties.AutomationId is added to editor controls

Future path for CI-driven UI tests:

  1. Add AutomationProperties.AutomationId incrementally to editor .axaml views
  2. Or use Avalonia's built-in headless test framework for programmatic UI testing without a display

Generated with Claude Code — Claude Opus 4.6 (1M context)

Clone this wiki locally