E2E: cleanup utility by phyllis-sy-wu · Pull Request #7171 · Shopify/cli

phyllis-sy-wu · 2026-04-02T21:33:42Z

WHY are these changes introduced?

E2E tests create apps on the Dev Dashboard that can accumulate over time — especially when tests fail mid-run, CI times out, or E2E_SKIP_CLEANUP=1 is used during development. This PR adds:

Standalone cleanup utility (scripts/cleanup.ts) — bulk cleanup of leftover apps via browser automation
E2E_SKIP_CLEANUP toggle — env var to disable per-test teardown for debugging

Naming convention: The E2E infra distinguishes two cleanup mechanisms:

teardownApp (in setup/app.ts, renamed in parent PRs) — per-test, automatic, runs in finally blocks, cleans up ONE app by name
cleanup.ts (in scripts/, this PR) — standalone script, manual, bulk operation, finds ALL apps matching a pattern on the dashboard

WHAT is this pull request doing?

New: `scripts/cleanup.ts`

Standalone cleanup script that finds E2E test apps on the Dev Dashboard, uninstalls them from all stores, and deletes them.

# Shorthand (from repo root):
pnpm test:e2e-cleanup              # Full cleanup: uninstall + delete
pnpm test:e2e-cleanup -- --list    # List matching apps without action
pnpm test:e2e-cleanup -- --delete  # Delete only (skip apps still installed)

# Or directly:
npx tsx packages/e2e/scripts/cleanup.ts              # Full cleanup: uninstall + delete
npx tsx packages/e2e/scripts/cleanup.ts --list        # List matching apps without action
npx tsx packages/e2e/scripts/cleanup.ts --uninstall   # Uninstall from all stores only (no delete)
npx tsx packages/e2e/scripts/cleanup.ts --delete      # Delete only (skip apps still installed on stores)
npx tsx packages/e2e/scripts/cleanup.ts --headed      # Show browser window
npx tsx packages/e2e/scripts/cleanup.ts --pattern X   # Match apps containing "X" (default: "E2E-")

Features:

Browser-based login (no PTY/node-pty dependency — works with npx tsx)
Reuses existing browser helpers from setup/app.ts and setup/browser.ts
Per-app retry with configurable max retries (default: 2)
Install status checking — skips delete for apps still installed on stores
30s action timeout (vs 60s in tests) — cleanup is dashboard navigation + clicks, shorter timeout means faster failure detection + retry
Progress UI with numbered apps, indented status lines, and summary
Exports cleanupAllApps() for use as a Playwright globalTeardown in the future

UI output:

[cleanup] Mode:    Uninstall + Delete
[cleanup] Org:     161686155
[cleanup] Pattern: "E2E-"

[cleanup] Logging in...
[cleanup] Logged in successfully.
[cleanup] Navigating to dashboard...
[cleanup] Found 3 app(s)

  1. E2E-deploy-1775107268211
  2. E2E-dev-1775107310343
  3. E2E-scaffold-1775107373607

[cleanup] [1/3] E2E-deploy-1775107268211
  Not installed
  Deleting...
  Deleted

[cleanup] [2/3] E2E-dev-1775107310343
  Uninstalling...
  Uninstalled
  Deleting...
  Deleted

[cleanup] [3/3] E2E-scaffold-1775107373607
  Uninstalling...
  Attempt 1 failed: Uninstall incomplete — some stores may remain
  Retry 1/2...
  Uninstalling...
  Uninstalled
  Deleting...
  Deleted

[cleanup] Complete: 3 succeeded

`E2E_SKIP_CLEANUP` toggle

All test finally blocks now check this env var. When set, both local file cleanup and remote app teardown are skipped:

# Skip all cleanup — inspect apps on Dev Dashboard + local files after test run
E2E_SKIP_CLEANUP=1 DEBUG=1 pnpm --filter e2e exec playwright test
# or 
E2E_SKIP_CLEANUP=1 pnpm e2e:test

How to test your changes?

Test the cleanup script:

Create some test apps: DEBUG=1 E2E_SKIP_CLEANUP=1 pnpm e2e:test app-deploy
List them: npx tsx packages/e2e/scripts/cleanup.ts --list
Full cleanup: npx tsx packages/e2e/scripts/cleanup.ts --headed
Verify apps are removed from dashboard

Test E2E_SKIP_CLEANUP:

Run with skip: E2E_SKIP_CLEANUP=1 pnpm e2e:test app-deploy
Verify app still exists on Dev Dashboard after test completes

Test error/retry handling:

To simulate failures and test the retry logic, temporarily add this in cleanup.ts before // Step 4:

// Simulate 500 errors — REMOVE AFTER TESTING
await page.route('**/apps/**/settings', (route) => {
  route.fulfill({status: 500, body: 'Internal Server Error'})
})

Examples

--list

[cleanup] Mode:    List only
[cleanup] Org:     161686155
[cleanup] Pattern: "E2E-"

[cleanup] Logging in...
[cleanup] Logged in successfully.
[cleanup] Navigating to dashboard...
[cleanup] Found 7 app(s)

  1. E2E-mcfg-def-1775163376879
  2. E2E-multi-cfg-1775163359813
  3. E2E-mcfg-def-1775162778046
  4. E2E-multi-cfg-1775162758320
  5. E2E-mcfg-def-1775161980433
  6. E2E-multi-cfg-1775161961138
  7. E2E-111

--uninstall (with retry)

[cleanup] Mode:    Uninstall only
[cleanup] Org:     161686155
[cleanup] Pattern: "E2E-"

[cleanup] Logging in...
[cleanup] Logged in successfully.
[cleanup] Navigating to dashboard...
[cleanup] Found 5 app(s)

  1. E2E-mcfg-def-1775162778046
  2. E2E-multi-cfg-1775162758320
  3. E2E-mcfg-def-1775161980433
  4. E2E-multi-cfg-1775161961138
  5. E2E-111

[cleanup] [1/5] E2E-mcfg-def-1775162778046
  Uninstalling...
  Uninstalled

[cleanup] [2/5] E2E-multi-cfg-1775162758320
  Uninstalling...
  Attempt 1 failed: Uninstall incomplete — some stores may remain
  Retry 1/2...
  Uninstalling...
  Uninstalled

[cleanup] [3/5] E2E-mcfg-def-1775161980433
  Not installed

[cleanup] [4/5] E2E-multi-cfg-1775161961138
  Uninstalling...
  Uninstalled

[cleanup] [5/5] E2E-111
  Not installed

[cleanup] Complete: 5 succeeded

--delete (with skips and failures)

[cleanup] Mode:    Delete only
[cleanup] Org:     161686155
[cleanup] Pattern: "E2E-"

[cleanup] Logging in...
[cleanup] Logged in successfully.
[cleanup] Navigating to dashboard...
[cleanup] Found 7 app(s)

  1. E2E-mcfg-def-1775163376879
  2. E2E-multi-cfg-1775163359813
  3. E2E-mcfg-def-1775162778046
  4. E2E-multi-cfg-1775162758320
  5. E2E-mcfg-def-1775161980433
  6. E2E-multi-cfg-1775161961138
  7. E2E-111

[cleanup] [1/7] E2E-mcfg-def-1775163376879
  Delete skipped (still installed)

[cleanup] [2/7] E2E-multi-cfg-1775163359813
  Delete skipped (still installed)

[cleanup] [3/7] E2E-mcfg-def-1775162778046
  Deleting...
  Attempt 1 failed: locator.scrollIntoViewIfNeeded: Timeout 15000ms exceeded.
  Retry 1/2...
  Deleting...
  Attempt 2 failed: locator.scrollIntoViewIfNeeded: Timeout 15000ms exceeded.
  Retry 2/2...
  Deleting...
  Failed: locator.scrollIntoViewIfNeeded: Timeout 15000ms exceeded.

...

[cleanup] Complete: 0 succeeded, 2 skipped, 5 failed

Full cleanup (default mode)

[cleanup] Mode:    Uninstall + Delete
[cleanup] Org:     161686155
[cleanup] Pattern: "E2E-"

[cleanup] Logging in...
[cleanup] Logged in successfully.
[cleanup] Navigating to dashboard...
[cleanup] Found 7 app(s)

  1. E2E-mcfg-def-1775163376879
  2. E2E-multi-cfg-1775163359813
  3. E2E-mcfg-def-1775162778046
  4. E2E-multi-cfg-1775162758320
  5. E2E-mcfg-def-1775161980433
  6. E2E-multi-cfg-1775161961138
  7. E2E-111

[cleanup] [1/7] E2E-mcfg-def-1775163376879
  Not installed
  Deleting...
  Deleted

[cleanup] [2/7] E2E-multi-cfg-1775163359813
  Not installed
  Deleting...
  Deleted

...

[cleanup] [7/7] E2E-111
  Not installed
  Deleting...
  Deleted

[cleanup] Complete: 7 succeeded

E2E_SKIP_CLEANUP=1 (apps remain on dashboard after test)

$ E2E_SKIP_CLEANUP=1 DEBUG=1 pnpm --filter e2e exec playwright test multi

  ✓  1 tests/multi-config-dev.spec.ts:14:3 › dev with -c flag loads the named config (19.3s)
  ✓  2 tests/multi-config-dev.spec.ts:90:3 › dev without -c flag uses default config (15.9s)
  2 passed (51.6s)

# Apps E2E-multi-cfg-* and E2E-mcfg-def-* remain on Dev Dashboard for inspection
# Clean up later with: npx tsx packages/e2e/scripts/cleanup.ts

Measuring impact

n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix

Checklist

I've considered possible cross-platform impacts (Mac, Linux, Windows)
I've considered possible documentation changes

phyllis-sy-wu · 2026-04-02T21:33:56Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

E2E: cleanup utility #7171 👈 (View in Graphite)
E2E: migrate tests to use new infra #7162
E2E: infra refactor + app tests update #7147
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2026-04-02T22:14:28Z

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

Some seemingly private modules might be re-exported through public modules.
If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/common/version.d.ts

@@ -1 +1 @@
-export declare const CLI_KIT_VERSION = "3.93.0";
\ No newline at end of file
+export declare const CLI_KIT_VERSION = "3.92.0";
\ No newline at end of file

This was referenced Apr 2, 2026

E2E: migrate tests to use new infra #7162

Open

E2E: infra refactor + app tests update #7147

Open

phyllis-sy-wu added the #gsd:49408 Agentic app validation label Apr 2, 2026

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch 2 times, most recently from 7ea8175 to d9e9bbf Compare April 2, 2026 22:12

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch 2 times, most recently from 5c6b425 to 7e992d2 Compare April 2, 2026 22:23

phyllis-sy-wu force-pushed the psyw-0401-E2E-migrate-remaining-tests-to-use-new-infra branch from 1d49241 to 8be6206 Compare April 2, 2026 22:24

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch 2 times, most recently from c84615f to b83a2b2 Compare April 2, 2026 23:53

E2E: cleanup utility

42157aa

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch from b83a2b2 to 42157aa Compare April 3, 2026 00:45

phyllis-sy-wu marked this pull request as ready for review April 3, 2026 00:48

phyllis-sy-wu requested a review from a team as a code owner April 3, 2026 00:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E: cleanup utility#7171

E2E: cleanup utility#7171
phyllis-sy-wu wants to merge 1 commit intopsyw-0401-E2E-migrate-remaining-tests-to-use-new-infrafrom
psyw-0402-E2E-cleanup-utility

phyllis-sy-wu commented Apr 2, 2026 •

edited

Loading

Uh oh!

phyllis-sy-wu commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

phyllis-sy-wu commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHY are these changes introduced?

WHAT is this pull request doing?

New: scripts/cleanup.ts

E2E_SKIP_CLEANUP toggle

How to test your changes?

Examples

Measuring impact

Checklist

Uh oh!

phyllis-sy-wu commented Apr 2, 2026

Uh oh!

github-actions bot commented Apr 2, 2026

Differences in type declarations

New type declarations

Existing type declarations

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

phyllis-sy-wu commented Apr 2, 2026 •

edited

Loading

New: `scripts/cleanup.ts`

`E2E_SKIP_CLEANUP` toggle