Skip to content

Commit f85530e

Browse files
test(perf): harden cloud Appium waits and performance selectors (#34008)
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## **Description** Backports BrowserStack-safe performance E2E hardening from `MMQA-2042-TestMU-PoC` into `main`, without bringing any TestMu AI provider, CI wiring, secrets, or PoC device-matrix changes. Cloud Appium visibility polls were paying an implicit-wait round-trip on every probe, which inflated measured durations. Perps and warm-start readiness also waited on fragile selectors. This PR: 1. Sets Appium implicit wait once per visibility poll (not per probe) in `PlaywrightAssertions` 2. Waits on the Perps withdraw control for market-list readiness and dismisses the tutorial before measuring 3. Asserts warm-start wallet readiness via the hamburger menu instead of the wallet container 4. Scrolls QuoteView “more networks” into view before tapping a network 5. Tightens Playwright performance grep to bare `@Performance` (`/@performance\b/`) Explicitly out of scope: TestMu provider/enricher/tunnel scripts, TestMu performance workflow jobs, `LT_*` / `TESTMU_*` secrets wiring, and Pixel 7 Pro PoC matrix mapping. ## **Changelog** CHANGELOG entry: null ## **Related issues** Refs: MMQA-2042 ## **Manual testing steps** ```gherkin Feature: performance E2E cloud wait hardening Scenario: warm start unlock reaches wallet Given a BrowserStack Android performance imported-wallet session When the warm-start login-to-wallet scenario unlocks the app Then the hamburger menu becomes visible within the step threshold And the measured duration is not inflated by per-probe implicit waits Scenario: Perps market list readiness Given a BrowserStack Android performance onboarding session with Perps When the Perps tutorial is dismissed if present Then the withdraw control is used as the market-list readiness signal And the select-market timer starts only after that control is visible Scenario: QuoteView network selection Given QuoteView is open and additional networks are below the fold When the test selects a network via Appium Then “more networks” is scrolled fully into view before the network tap ``` N/A for interactive app QA beyond BrowserStack performance suites above — this PR only changes test/CI helper code. ## **Screenshots/Recordings** N/A — test-infrastructure and CI setup changes only; no end-user UI changes. ### **Before** N/A ### **After** N/A ## **Pre-merge author checklist** - [x] I've followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [x] I've completed the PR template to the best of my ability - [x] I've included tests if applicable - [x] I've documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [ ] I've applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. #### Performance checks (if applicable) - [ ] I've tested on Android - Ideally on a mid-range device; emulator is acceptable - [ ] I've tested with a power user scenario - Use these [power-user SRPs](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/edit-v2/401401446401?draftShareId=9d77e1e1-4bdc-4be1-9ebb-ccd916988d93) to import wallets with many accounts and tokens - [ ] I've instrumented key operations with Sentry traces for production performance metrics - See [`trace()`](/app/util/trace.ts) for usage and [`addToken`](/app/components/Views/AddAsset/components/AddCustomToken/AddCustomToken.tsx#L274) for an example For performance guidelines and tooling, see the [Performance Guide](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/400085549067/Performance+Guide+for+Engineers). ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. ## **Test plan** - [ ] CI status checks pass on this PR - [ ] Diff contains only the 7 intended files (no TestMu / `LT_*` / `TESTMU_*` wiring) - [ ] Spot-check BrowserStack Android performance onboarding + imported-wallet (warm-start, Perps, network selection) <!-- CURSOR_AGENT_PR_BODY_END --> <div><a href="https://cursor.com/agents/bc-18cd73a7-d8af-42b0-ba8b-9ed0c1c62b2a"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/background-agent?bcId=bc-18cd73a7-d8af-42b0-ba8b-9ed0c1c62b2a"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>
1 parent 99313f9 commit f85530e

15 files changed

Lines changed: 332 additions & 119 deletions

tests/framework/Constants.ts

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,24 @@ export const DEFAULT_ACTION_TIMEOUT_MS = 5_000;
105105
/** WebDriver HTTP timeout for BrowserStack session creation (grid can take several minutes). */
106106
export const DEFAULT_BROWSERSTACK_CONNECTION_RETRY_TIMEOUT_MS = 300_000;
107107

108+
/**
109+
* WDIO HTTP request retries for BrowserStack hub calls (including POST /session).
110+
* Busy grids often abort session creation once; a higher count recovers without
111+
* burning a full Playwright test retry.
112+
*/
113+
export const DEFAULT_BROWSERSTACK_CONNECTION_RETRY_COUNT = 5;
114+
115+
/**
116+
* Outer attempts in BrowserStackProvider.getDriver() when session creation fails
117+
* with a transient hub/timeout error after WDIO's own connection retries.
118+
* This is the session-only retry path — Playwright test retries stay at the
119+
* shared CI default and are not raised for product/threshold failures.
120+
*/
121+
export const DEFAULT_BROWSERSTACK_SESSION_CREATE_MAX_ATTEMPTS = 5;
122+
123+
/** Backoff between BrowserStackProvider session-create attempts, in ms. */
124+
export const DEFAULT_BROWSERSTACK_SESSION_CREATE_RETRY_DELAY_MS = 5_000;
125+
108126
/** BrowserStack maximum allowed idle timeout between WebDriver commands, in seconds. */
109127
export const DEFAULT_BROWSERSTACK_IDLE_TIMEOUT_SECONDS = 300;
110128

tests/framework/PlaywrightAssertions.ts

Lines changed: 35 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,10 @@ export default class PlaywrightAssertions {
4444
* (`isExisting`). This avoids the multiple internal HTTP round-trips that
4545
* WebdriverIO's `waitForDisplayed` performs on each iteration.
4646
*
47+
* Implicit wait is set once for the whole poll (not per attempt) so cloud
48+
* Appium providers do not pay 2× `setTimeout` RTT every probe — that
49+
* overhead was inflating performance timers.
50+
*
4751
* When measuring, only the successful confirm (+ probe) counts as infra.
4852
* A shorter poll interval reduces detection lag without zeroing app time.
4953
*/
@@ -62,31 +66,41 @@ export default class PlaywrightAssertions {
6266
? this.POLL_INTERVAL_WHILE_MEASURING_MS
6367
: this.POLL_INTERVAL_MS;
6468
const start = Date.now();
65-
while (Date.now() - start < timeout - this.FINAL_WAIT_RESERVE_MS) {
66-
const remaining = timeout - (Date.now() - start);
67-
if (remaining <= 0) {
68-
break;
69-
}
70-
const t0 = Date.now();
71-
try {
72-
const exists = await withImplicitWait(this.POLL_IMPLICIT_WAIT_MS, () =>
73-
el.unwrap().isExisting(),
74-
);
75-
if (exists) {
76-
const displayed = await el.isVisible();
77-
if (displayed) {
78-
if (tracking) {
79-
recordSuccessPollCommand(Date.now() - t0);
80-
await this.probeOverhead(el);
69+
70+
const found = await withImplicitWait(
71+
this.POLL_IMPLICIT_WAIT_MS,
72+
async () => {
73+
while (Date.now() - start < timeout - this.FINAL_WAIT_RESERVE_MS) {
74+
const remaining = timeout - (Date.now() - start);
75+
if (remaining <= 0) {
76+
break;
77+
}
78+
const t0 = Date.now();
79+
try {
80+
const exists = await el.unwrap().isExisting();
81+
if (exists) {
82+
const displayed = await el.isVisible();
83+
if (displayed) {
84+
if (tracking) {
85+
recordSuccessPollCommand(Date.now() - t0);
86+
await this.probeOverhead(el);
87+
}
88+
return true;
89+
}
8190
}
82-
return;
91+
} catch {
92+
// element not ready yet
8393
}
94+
await sleep(Math.min(interval, remaining));
8495
}
85-
} catch {
86-
// element not ready yet
87-
}
88-
await sleep(Math.min(interval, remaining));
96+
return false;
97+
},
98+
);
99+
100+
if (found) {
101+
return;
89102
}
103+
90104
const remainingTimeout = timeout - (Date.now() - start);
91105
await el.waitForDisplayed({
92106
timeout: Math.max(interval, remainingTimeout),

tests/framework/services/providers/browserstack/BrowserStackConfigBuilder.ts

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import path from 'path';
33
import type { BrowserStackConfig } from '../../../types';
44
import type { ProjectConfig } from '../../common/types';
55
import {
6+
DEFAULT_BROWSERSTACK_CONNECTION_RETRY_COUNT,
67
DEFAULT_BROWSERSTACK_CONNECTION_RETRY_TIMEOUT_MS,
78
DEFAULT_BROWSERSTACK_IDLE_TIMEOUT_SECONDS,
89
DEFAULT_BROWSERSTACK_NEW_COMMAND_TIMEOUT_SECONDS,
@@ -79,7 +80,8 @@ export class BrowserStackConfigBuilder {
7980
: DEFAULT_BROWSERSTACK_CONNECTION_RETRY_TIMEOUT_MS;
8081

8182
logger.info(
82-
`BrowserStack WebDriver connectionRetryTimeout: ${connectionRetryTimeout}ms`,
83+
`BrowserStack WebDriver connectionRetryTimeout: ${connectionRetryTimeout}ms, ` +
84+
`connectionRetryCount: ${DEFAULT_BROWSERSTACK_CONNECTION_RETRY_COUNT}`,
8385
);
8486
logger.info(
8587
`BrowserStack idleTimeout: ${DEFAULT_BROWSERSTACK_IDLE_TIMEOUT_SECONDS}s, newCommandTimeout: ${DEFAULT_BROWSERSTACK_NEW_COMMAND_TIMEOUT_SECONDS}s`,
@@ -98,7 +100,7 @@ export class BrowserStackConfigBuilder {
98100
hostname: 'hub.browserstack.com',
99101
// Default webdriver is 120s; BS session POST often exceeds that on busy grids.
100102
connectionRetryTimeout,
101-
connectionRetryCount: 3,
103+
connectionRetryCount: DEFAULT_BROWSERSTACK_CONNECTION_RETRY_COUNT,
102104
capabilities: {
103105
'bstack:options': {
104106
debug: true,

tests/framework/services/providers/browserstack/BrowserStackProvider.ts

Lines changed: 95 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,57 @@
11
import { remote, type Browser } from 'webdriverio';
22
import { BaseServiceProvider } from '../../common/base/BaseServiceProvider.ts';
33
import type { ProjectConfig } from '../../common/types.ts';
4+
import {
5+
DEFAULT_BROWSERSTACK_SESSION_CREATE_MAX_ATTEMPTS,
6+
DEFAULT_BROWSERSTACK_SESSION_CREATE_RETRY_DELAY_MS,
7+
} from '../../../Constants.ts';
48
import { BrowserStackAPI } from './BrowserStackAPI.ts';
59
import { BrowserStackConfigBuilder } from './BrowserStackConfigBuilder.ts';
610

11+
/**
12+
* Only retry busy-grid / transport flakes. Do NOT match generic WDIO session
13+
* text like "Failed to create a session" or "wd/hub/session" — those also
14+
* appear for permanent failures (bad credentials, invalid app URL, caps).
15+
*/
16+
const TRANSIENT_SESSION_ERROR_PATTERNS = [
17+
'aborted due to timeout',
18+
'operation was aborted',
19+
'ECONNRESET',
20+
'ETIMEDOUT',
21+
'ECONNREFUSED',
22+
'socket hang up',
23+
'network timeout',
24+
'All parallel tests are currently in use',
25+
'All devices are busy',
26+
'DEVICE_QUEUE_TIMEOUT',
27+
] as const;
28+
29+
const PERMANENT_SESSION_ERROR_PATTERNS = [
30+
'Invalid username or password',
31+
'Authentication failed',
32+
'Unauthorized',
33+
'App not found',
34+
'Invalid app',
35+
'app_url',
36+
'BROWSERSTACK_USERNAME',
37+
'BROWSERSTACK_ACCESS_KEY',
38+
'buildPath is required',
39+
] as const;
40+
41+
function isTransientBrowserStackSessionError(error: unknown): boolean {
42+
const message = error instanceof Error ? error.message : String(error);
43+
if (
44+
PERMANENT_SESSION_ERROR_PATTERNS.some((pattern) =>
45+
message.includes(pattern),
46+
)
47+
) {
48+
return false;
49+
}
50+
return TRANSIENT_SESSION_ERROR_PATTERNS.some((pattern) =>
51+
message.includes(pattern),
52+
);
53+
}
54+
755
/**
856
* Service provider for BrowserStack cloud testing
957
*/
@@ -25,7 +73,9 @@ export class BrowserStackProvider extends BaseServiceProvider {
2573
}
2674

2775
/**
28-
* Create and return WebDriver browser instance for BrowserStack
76+
* Create and return WebDriver browser instance for BrowserStack.
77+
* Retries transient hub/session timeouts so a single busy-grid abort does
78+
* not consume the whole Playwright test retry budget.
2979
*/
3080
async getDriver(): Promise<Browser> {
3181
this.logger.info(
@@ -34,16 +84,52 @@ export class BrowserStackProvider extends BaseServiceProvider {
3484

3585
const configBuilder = new BrowserStackConfigBuilder(this.project);
3686
const config = configBuilder.build();
37-
87+
const maxAttempts = DEFAULT_BROWSERSTACK_SESSION_CREATE_MAX_ATTEMPTS;
3888
const sessionCreationStart = Date.now();
39-
const browser = await remote(config);
40-
this.sessionCreationDurationMs = Date.now() - sessionCreationStart;
41-
this.sessionId = browser.sessionId;
89+
let lastError: unknown;
4290

43-
this.logger.info(
44-
`Driver created for BrowserStack with session: ${this.sessionId} (session creation took ${this.sessionCreationDurationMs}ms)`,
45-
);
46-
return browser;
91+
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
92+
try {
93+
const browser = await remote(config);
94+
this.sessionCreationDurationMs = Date.now() - sessionCreationStart;
95+
this.sessionId = browser.sessionId;
96+
97+
this.logger.info(
98+
`Driver created for BrowserStack with session: ${this.sessionId} ` +
99+
`(session creation took ${this.sessionCreationDurationMs}ms` +
100+
(attempt > 1 ? `, attempt ${attempt}/${maxAttempts}` : '') +
101+
`)`,
102+
);
103+
return browser;
104+
} catch (error) {
105+
lastError = error;
106+
const message = error instanceof Error ? error.message : String(error);
107+
const isTransient = isTransientBrowserStackSessionError(error);
108+
109+
if (!isTransient || attempt === maxAttempts) {
110+
this.logger.error(
111+
`BrowserStack session creation failed ` +
112+
`(attempt ${attempt}/${maxAttempts}` +
113+
`${isTransient ? ', transient' : ''}): ${message}`,
114+
);
115+
throw error;
116+
}
117+
118+
this.logger.warn(
119+
`BrowserStack session creation failed transiently ` +
120+
`(attempt ${attempt}/${maxAttempts}); retrying in ` +
121+
`${DEFAULT_BROWSERSTACK_SESSION_CREATE_RETRY_DELAY_MS}ms: ${message}`,
122+
);
123+
await new Promise((resolve) =>
124+
setTimeout(
125+
resolve,
126+
DEFAULT_BROWSERSTACK_SESSION_CREATE_RETRY_DELAY_MS,
127+
),
128+
);
129+
}
130+
}
131+
132+
throw lastError instanceof Error ? lastError : new Error(String(lastError));
47133
}
48134

49135
/**

tests/page-objects/Onboarding/CreatePasswordView.ts

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -314,17 +314,20 @@ class CreatePasswordView {
314314
private async readMarketingCheckboxCheckedAppium(
315315
checkbox: Awaited<ReturnType<typeof asPlaywrightElement>>,
316316
): Promise<boolean | undefined> {
317-
try {
318-
const attributes: Record<string, unknown> = {
319-
'aria-checked': await checkbox.getAttribute('aria-checked'),
320-
checked: await checkbox.getAttribute('checked'),
321-
value: await checkbox.getAttribute('value'),
322-
};
323-
324-
return this.parseMarketingCheckboxCheckedState(attributes);
325-
} catch {
326-
return undefined;
317+
// Fetch attributes independently. Android UiAutomator2 does not support
318+
// `aria-checked`; requesting it first used to abort the whole read and
319+
// leave isChecked undefined (see seedless Apple/Telegram perf failures).
320+
const attributes: Record<string, unknown> = {};
321+
322+
for (const name of ['checked', 'value', 'aria-checked'] as const) {
323+
try {
324+
attributes[name] = await checkbox.getAttribute(name);
325+
} catch {
326+
// Unsupported on this platform/driver — try the next candidate.
327+
}
327328
}
329+
330+
return this.parseMarketingCheckboxCheckedState(attributes);
328331
}
329332

330333
/**

tests/page-objects/Perps/PerpsMarketListView.ts

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import {
22
PerpsHomeViewSelectorsIDs,
3+
PerpsMarketBalanceActionsSelectorsIDs,
34
PerpsMarketListViewSelectorsIDs,
45
PerpsMarketRowItemSelectorsIDs,
56
PerpsTokenSelectorSelectorsIDs,
@@ -141,6 +142,12 @@ class PerpsMarketListView {
141142
return Matchers.getElementByID(PerpsTokenSelectorSelectorsIDs.CLOSE_BUTTON);
142143
}
143144

145+
get withdrawButton() {
146+
return Matchers.getElementByID(
147+
PerpsMarketBalanceActionsSelectorsIDs.WITHDRAW_BUTTON,
148+
);
149+
}
150+
144151
// Actions
145152
async tapMarketRowItemBTC() {
146153
await encapsulatedAction({

tests/page-objects/swaps/QuoteView.ts

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -465,6 +465,18 @@ class QuoteView {
465465
});
466466
},
467467
appium: async () => {
468+
// Best-effort only: some swap flows never expose "more networks", and
469+
// forcing scrollIntoView there fails Appium smoke after 30 scrolls.
470+
try {
471+
const moreNetworks = await asPlaywrightElement(
472+
this.moreNetworksButton,
473+
);
474+
if (await moreNetworks.unwrap().isExisting()) {
475+
await PlaywrightGestures.scrollIntoViewFullyVisible(moreNetworks);
476+
}
477+
} catch {
478+
// Continue — the target network may already be visible without this control.
479+
}
468480
const networkElement =
469481
await PlaywrightMatchers.getElementByCatchAll(network);
470482
await PlaywrightAssertions.expectElementToBeVisible(networkElement, {

tests/page-objects/wallet/AccountListBottomSheet.ts

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,8 +54,11 @@ class AccountListBottomSheet {
5454
),
5555
appium: () =>
5656
PlatformDetector.isIOS()
57-
? PlaywrightMatchers.getElementByText(
57+
? // Exact match: contains("Accounts") also hits "Connect accounts" /
58+
// "Edit accounts" nodes that can exist while displayed=false.
59+
PlaywrightMatchers.getElementByText(
5860
AccountListBottomSheetSelectorsText.ACCOUNTS_LIST_TITLE,
61+
true,
5962
)
6063
: PlaywrightMatchers.getElementById(
6164
AccountListBottomSheetSelectorsIDs.ACCOUNT_LIST_ID,
@@ -318,6 +321,7 @@ class AccountListBottomSheet {
318321
el = PlatformDetector.isIOS()
319322
? await PlaywrightMatchers.getElementByText(
320323
AccountListBottomSheetSelectorsText.ACCOUNTS_LIST_TITLE,
324+
true,
321325
)
322326
: await PlaywrightMatchers.getElementById(
323327
AccountListBottomSheetSelectorsIDs.ACCOUNT_LIST_ID,

tests/performance/device-matrix.json

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,10 @@
11
{
22
"android_devices": [
3-
{
4-
"name": "Samsung Galaxy S25 Ultra",
5-
"os_version": "15.0",
6-
"category": "high",
7-
"description": "High-end Samsung flagship device with Android 15"
8-
},
93
{
104
"name": "Google Pixel 8 Pro",
115
"os_version": "14.0",
126
"category": "low",
13-
"description": "High-end Google flagship device with Android 14"
7+
"description": "Google Pixel 8 Pro with Android 14 — sole Android device for performance E2E (Galaxy removed to reduce cost/flakes)"
148
}
159
],
1610
"ios_devices": [
@@ -33,11 +27,11 @@
3327
"low": "Low-end devices for backward compatibility and emerging market testing"
3428
},
3529
"os_coverage": {
36-
"android": ["13.0", "14.0"],
30+
"android": ["14.0"],
3731
"ios": ["16", "17"]
3832
},
3933
"notes": {
4034
"ios_limitations": "iPhone 11 (iOS 13) and iPhone 8 (iOS 11) have OS versions not supported by MetaMask app",
41-
"android_performance": "TECNO KI5k included despite low-end specs due to high production volume (250.2K units) in emerging markets"
35+
"android_performance": "Android performance E2E runs on Google Pixel 8 Pro only (Samsung Galaxy removed from the matrix)."
4236
}
4337
}

tests/performance/login/launch-times/warm-start-login-to-wallet.spec.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ perfTest.describe(
6666
await LoginView.tapLoginButton();
6767
await timer1.measure(async () => {
6868
await PlaywrightAssertions.expectElementToBeVisible(
69-
asPlaywrightElement(WalletView.container),
69+
asPlaywrightElement(WalletView.hamburgerMenuButton),
7070
{
7171
description: 'Wallet balance should be visible',
7272
timeout: 30 * 1000, // 30 seconds

0 commit comments

Comments
 (0)