GUI Validation Methods

GUI Validation Methods

Comparison of two approaches for validating the Avalonia GUI: MCP Computer Use (visual screenshot-based) vs PowerShell UI Automation (structural element tree).

Test Setup

• Target view: Unit Editor — Eirika (Unit 01) in FE8U
• App: FEBuilderGBA Avalonia (Debug build, .NET 9.0)
• Screen: 3240x2160 (Windows 11, 200% DPI scaling)
• Date: 2026-03-23

Approach 1: MCP Computer Use

The MCP Computer Use server (tools/mcp-computer-use/) provides screenshot capture, mouse click, keyboard input, and window management via JSON-RPC over stdio.

What it can validate

Aspect	Result
Editor opened correctly	"Unit Editor" title visible with address 0x00803D64
Correct unit selected	Eirika portrait rendered (teal-haired character sprite)
Identity fields	Name ID=530, Name="Eirika", Unit ID=1, Class="02 Lord", Portrait=2
Base stats	LV=1, Lck=5, all other stats visible and readable
Weapon levels	Sword=1 (rank E), all weapon types displayed
Visual layout	Sections (Identity, Base Stats, Weapon Levels) properly separated
Portrait rendering	Sprite image renders correctly with proper colors
Navigation	Unit list on left with portrait thumbnails, scrollable

Coordinate mapping

Screenshots are captured at full screen resolution (3240x2160) and scaled to max 1200x800.

• Scale factor: 2.7x
• Click coordinates: screenshot pixel position x 2.7 = screen coordinates
• Tip: Always focus_window before clicking — the terminal steals focus

Strengths

• Zero setup beyond existing MCP server
• Validates what the user actually sees
• Catches visual bugs (wrong colors, misaligned layout, missing images)
• Works for all 357 Avalonia views without any code changes

Known Issues

• Window focus is unreliable: focus_window often fails with "Access denied" or "No error message available" when the terminal (Claude Code) holds foreground lock. Clicking the window title bar sometimes helps, but keyboard input may still route to the terminal instead of the Avalonia app.
• High-DPI coordinate mismatch: On 200%+ DPI displays, the window rect reported by find_window is in physical pixels while click/type_text operate in logical pixels. The 2.7x scale factor must be applied manually, and miscalculation causes clicks to miss targets.
• Black screenshots: screenshot can return all-black images when the Avalonia window is behind other windows or on a secondary monitor. focus_window does not reliably bring it to front.
• Overall flakiness: Due to the above, MCP GUI validation is not reliable enough for CI-blocking proof. It works best as a manual-assist tool where a developer can intervene when focus/DPI issues occur. For automated proof, export tile sheet / frame PNGs programmatically instead.

Limitations

• Pixel-level comparison can be flaky with font rendering differences
• Cannot assert exact numeric values without OCR (rely on visual inspection)
• Slower than structural queries (screenshot + network round-trip)
• Flaky on high-DPI / multi-window setups — see Known Issues above

Approach 2: PowerShell UI Automation

Uses System.Windows.Automation (UIAutomationClient) to inspect the Avalonia window's automation element tree.

What it found

=== WINDOW FOUND ===
Name: FEBuilderGBA - FE8U.gba
ClassName: MainWindow
BoundingRect: 405,450,3236,2119

=== TOTAL DESCENDANTS: 279 ===
=== ELEMENTS WITH AUTOMATION IDs: 40 ===

Element Type	Count	What's visible
Text elements	34	Status bar, expander headers only
Edit fields	1	Filter textbox only
Combo boxes	0	None
List items	0	None
Buttons with AutomationId	34	Expander headers (Characters, Items, Maps...)

What it CANNOT see

The entire Unit Editor content is invisible to UI Automation:

• No edit fields (Name ID, Unit ID, stats — all missing)
• No combo boxes (Class ID, Affinity dropdowns — missing)
• No list items (unit list with portraits — missing)
• No images (portrait rendering — impossible)

Why

1. Avalonia's AutomationPeer support is opt-in — controls without explicit automation peers are invisible
2. Editor views don't set AutomationProperties.AutomationId on their controls
3. Dynamically-loaded user controls inside ContentControl areas aren't exposed to the automation tree
4. Only the main window shell (menu, status bar, expander headers) has automation support

Strengths

• Deterministic — no visual comparison needed
• Can read exact text values programmatically
• Scriptable for CI/CD pipelines (headless capable)
• Fast queries once the tree is built

Known Issues

• Window focus is unreliable: focus_window often fails with "Access denied" or "No error message available" when the terminal (Claude Code) holds foreground lock. Clicking the window title bar sometimes helps, but keyboard input may still route to the terminal instead of the Avalonia app.
• High-DPI coordinate mismatch: On 200%+ DPI displays, the window rect reported by find_window is in physical pixels while click/type_text operate in logical pixels. The 2.7x scale factor must be applied manually, and miscalculation causes clicks to miss targets.
• Black screenshots: screenshot can return all-black images when the Avalonia window is behind other windows or on a secondary monitor. focus_window does not reliably bring it to front.
• Overall flakiness: Due to the above, MCP GUI validation is not reliable enough for CI-blocking proof. It works best as a manual-assist tool where a developer can intervene when focus/DPI issues occur. For automated proof, export tile sheet / frame PNGs programmatically instead.

Limitations

• Blind to all editor content in current state
• Requires adding AutomationProperties.AutomationId to hundreds of controls across 357 views
• Avalonia's UI Automation is less mature than WinForms

Side-by-Side Comparison

Validation Target	MCP Computer Use	PowerShell UI Automation
Unit Editor opened	Screenshot shows title + address	Cannot detect
Eirika selected	Portrait visible, name readable	Cannot detect (0 list items)
Name ID = 530	Visible in screenshot	Cannot detect (0 edit fields)
Class = 02 Lord	Visible in combo box	Cannot detect (0 combo boxes)
Base Stats (LV 1, Lck 5)	Visible in stat fields	Cannot detect
Portrait renders correctly	Teal-haired sprite visible	Impossible (no visual)
Layout/spacing correct	Sections properly separated	Impossible
Status bar text	N/A	`FE8U.gba
Menu structure	Visible in screenshot	34 expander headers enumerable

Recommendation

Use MCP Computer Use as a manual-assist GUI validation tool, with programmatic PNG export as fallback proof.

• Most validations are visual (portrait rendering, map display, color accuracy)
• Works out of the box with zero code changes to Avalonia views
• Validates the actual user experience, not just the element tree
• Already configured in .mcp.json and ready to use
• Caveat: Flaky on high-DPI displays and multi-window setups — window focus and keyboard routing are unreliable. When MCP cannot capture the Avalonia window, use programmatic PNG export (e.g., IImage.Save()) as screenshot proof instead.

Use PowerShell UI Automation only for:

• Status bar assertions (ROM version, file size)
• Main menu structure verification (expander presence)
• Future: if AutomationProperties.AutomationId is added to editor controls

Future path for CI-driven UI tests: (tracked in #243)

1. Add AutomationProperties.AutomationId incrementally to editor .axaml views
2. Or use Avalonia's built-in headless test framework for programmatic UI testing without a display

---

Generated with Claude Code — Claude Opus 4.6 (1M context)