Cut Chrome/Firefox/iPhone/iPad UI tests over to Device Farm

[davidsbailey]

Starts running UI tests in Device Farm alongside SauceLabs within rake test:ui_all. Renames the SauceLabs UI task regular_ui to saucelabs_ui, drops its parallelism from 50 to 15, and narrows it to -c Safari since Chrome/Firefox/iPhone/iPad now run on Device Farm. Removes the unused devicefarm_ui combined task.

Reframes the user-visible labeling so oncall sees suites by the browsers they cover, not the provider:

• saucelabs_ui => "Safari UI"
• devicefarm_desktop_ui => "Chrome + Firefox UI"
• devicefarm_mobile_ui => "Mobile UI"
• eyes_ui => "Eyes"

slack message and status page prefixes also now derive from the active browser configs (Safari, Chrome + Firefox, etc) instead of the device_farm flag, making it easier to find test failures for a certain browser without having to remember whether it runs on SauceLabs or Device Farm. The cross-page navigation row also now links to all 4 pages.

The _df disambiguator in the _output.html filenames is no longer needed and has been removed.

Screenshots

status page navigation

Screen.Recording.2026-05-05.at.2.42.34.PM.mov

slack output

Testing story

• tested locally against test-studio.code.org and just one feature file (see screen recording) via locally modified version of this command: Dave-M4:~/src/cdo (test *)$ bundle exec rake test:ui_all
  • clicked through resulting status pages to verify status page nav linkage, rerun cmds, and saucelabs + device farm links in cucumber logs.
• after merging Revamp UI test status pages #72453, I ran rake test:devicefarm_ui on the test machine. All Device Farm Desktop UI tests passed (slack). 2 Mobile UI tests failed (slack), but then passed after a total of 3 reruns.

Deployment notes

to do on the test machine after DTT:

• verify linkage between new status pages
• remove old test_status*.html pages, to ensure no one is relying on old bookmarks
• run bin/reset_flakiness, to clear out the slew of errors from multiple saucelabs problems in the past week

[davidsbailey]

tagging @sureshc for review. heads up @cat5inthecradle that this forces us onto latest browser versions for Chrome and Firefox, since Device Farm only supports latest through latest minus 2. Mobile is more complicated -- we run on a pool of devices in which are all on iOS 18 or higher, but could be a different model, screen size and/or iOS version on each run.

[sureshc]

Could we add some clarity and explain that we're running macOS Safari in SauceLabs, and mobile (iOS/iPadOS) Safari in Device Farm? There are a bunch of places where we could replace "Safari UI" with "macOS Safari UI" and "Mobile UI" with "Mobile Safari UI", so not sure it's worth making this type of change when we have a more pressing goal of completing this transition quickly.

Suggested change

	| Safari UI | SauceLabs | `Safari` (`browsers_saucelabs.json`) |
	| Chrome + Firefox UI | AWS Device Farm | `Chrome`, `Firefox` (`browsers_device_farm.json`) |
	| Mobile UI | AWS Device Farm | `iPhone`, `iPad` (`browsers_device_farm.json`) |
	| Eyes | SauceLabs | `Chrome`, `iPhone` (Applitools visual diff, `@eyes` / `@eyes_mobile`) |
	| macOS Safari UI | SauceLabs | `macOS Safari` (`browsers_saucelabs.json`) |
	| Chrome + Firefox UI | AWS Device Farm | `Chrome`, `Firefox` (`browsers_device_farm.json`) |
	| Mobile Safari UI | AWS Device Farm | `iPhone Safari`, `iPad Safari` (`browsers_device_farm.json`) |
	| Eyes | SauceLabs | `Chrome`, `iPhone` (Applitools visual diff, `@eyes` / `@eyes_mobile`) |

[davidsbailey]

Thank you for the feedback Suresh! I've added some clarifying info to the "Browsers" column to clarify macOS, Windows, and Safari as needed.

I haven't changed the Suite name here because I want to keep it consistent with everywhere else we show the suite name (slack messages, status pages), and I don't want to clutter up those places with extra wording just yet.

I suspect that most of the team will intuit that an unqualified Safari means desktop since that's been the convention on the actual browser config name for quite a while now, but if there turns out to be confusion about it then I'd be happy to add more wording at that time.