E2E: utility - cleanup apps

[phyllis-sy-wu]

WHY are these changes introduced?

E2E tests create apps that can accumulate when tests fail mid-run, CI times out, or teardown fails. This script automates bulk-clean for leftover apps.

WHAT is this pull request doing?

cleanup-apps.ts

Standalone cleanup script that finds leftover E2E test apps on the Dev Dashboard, uninstalls them from all stores, and deletes them.

pnpm --filter e2e exec tsx scripts/cleanup-apps.ts              # Full cleanup: uninstall + delete
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --list        # List matching apps with install counts
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --uninstall   # Uninstall from all stores only (no delete)
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --delete      # Delete only apps with 0 installs
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed      # Show browser window
pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --pattern X   # Match apps containing "X" (default: "E2E-")

Logic

Discovery phase:

1. Log in via completeLogin helper
2. Navigate to Dev Dashboard (retry up to 3 times on 500/502 — hard-fail after 3 attempts instead of silently continuing)
3. Find app cards via a[href*="/apps/"] selectors
4. Extract app name from card text (split on install count pattern), install count via regex, and URL from href
5. Filter by name pattern (default: E2E-)
6. Paginate via a[href*="next_cursor"] and repeat 3–5 on each page

Uninstall (per app):

1. Navigate to {appUrl}/installs
2. Collect store slugs via FQDN regex on full page HTML (page.content())
3. If no FQDNs found on this page: fall back to table row text (store name = slug)
4. Paginate via button#nextURL and repeat 2–3 on each page
5. For each store slug:
   a. Navigate to admin.shopify.com/store/{slug}/settings/apps
   b. Dismiss Dev Console if visible
   c. Find app by name via span:has-text("{appName}"):not([class*="Polaris"])
   d. If not visible → already uninstalled, skip to next store
   e. Click ⋯ menu button (next sibling) → click "Uninstall" → click confirm
   f. Verify: reload page, check the app span is no longer visible
   g. If still visible → mark allUninstalled = false
   h. On error: log store slug and error message, mark allUninstalled = false
6. Final verification: navigate back to {appUrl}/installs, scan every row across all pages
7. If any non-empty row found on any page → return false
8. Return allUninstalled

Delete (per app):

1. Navigate to {appUrl}/settings
2. Check for error pages: 404 → app already deleted (return true); 500/502 → throw to retry
3. Find "Delete app" button, scroll into view — if isEnabled() returns false, reload once as a fallback for propagation lag; click({timeout: 2 * BROWSER_TIMEOUT.long})'s auto-wait handles the rest at ~100ms granularity
4. Type "DELETE" in confirmation input (if present)
5. Click confirm "Delete app" button
6. Verify: reload {appUrl}/settings, check for 404 (success) or 500/502 (throw to retry)
7. Up to 3 outer attempts per app

Per-app retry wrapper:

1. Each app gets up to maxRetries + 1 attempts (default: 3 total)
2. On failure: log error, wait, re-navigate to dashboard, retry from uninstall
3. Per-app elapsed time printed ((Xs)); summary printed at end: X succeeded, Y skipped, Z failed (Xs total)

--list mode: runs discovery only, prints app names and install counts.

--uninstall mode: runs uninstall only, skips apps with 0 installs.

--delete mode: runs delete only, skips apps with installs > 0.

Features:

• Dashboard error handling: hard-fails after 3 consecutive 500/502 responses
• Store slug extraction via full HTML regex with per-page table text fallback
• Per-store uninstall verification (reload + check)
• Per-store uninstall error logging (store slug + error message)
• Paginated final installs verification across all pages
• Error page differentiation: 404 = already deleted (success), 500/502 = server error (retry)
• Exports cleanupAllApps() for use as a Playwright globalTeardown or from other scripts

How is this different from per-test teardown?

• Per-test teardown (setup/teardown.ts) — knows the specific app name and store FQDN, uses direct URLs, no discovery. Runs automatically in test finally blocks.
• cleanup-apps.ts (bulk, manual) — discovers all matching apps via dashboard pagination, discovers stores via installs page. Safety net for orphaned apps from failed/interrupted test runs.

How to test your changes?

1. Create leftover apps by skipping cleanup:

   E2E_SKIP_TEARDOWN=1 DEBUG=1 pnpm --filter e2e exec playwright test app

2. List them:

   pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --list

3. Clean up:

   pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed

Example

pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed

cleanup-apps.mov

Expand for complete log

pnpm --filter e2e exec tsx scripts/cleanup-apps.ts --headed
[cleanup-apps] Mode:    Uninstall + Delete
[cleanup-apps] Org:     161686155
[cleanup-apps] Pattern: "E2E-"
[cleanup-apps] Logging in...
[cleanup-apps] Logged in successfully.
[cleanup-apps] Navigating to dashboard...
[cleanup-apps] Dashboard loaded.
[cleanup-apps] Finding matching apps...
[cleanup-apps] Found 4 app(s)
  1. E2E-scaffold-1776750657171 (0 installs)
  2. E2E-dev-1776750673263 (1 install)
  3. E2E-deploy-1776750657171 (0 installs)
  4. E2E-ext-only-1776750657176 (0 installs)
[cleanup-apps] [1/4] E2E-scaffold-1776750657171
  Not installed
  Deleting...
  Deleted
  (3.2s)
[cleanup-apps] [2/4] E2E-dev-1776750673263
  Uninstalling...
  Uninstalled
  Deleting...
  Deleted
  (8.7s)
[cleanup-apps] [3/4] E2E-deploy-1776750657171
  Not installed
  Deleting...
  Deleted
  (3.1s)
[cleanup-apps] [4/4] E2E-ext-only-1776750657176
  Not installed
  Deleting...
  Deleted
  (3.3s)

[cleanup-apps] Complete: 4 succeeded (18.4s total)

Post-release steps

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

[phyllis-sy-wu]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• E2E - QA: init, deploy, versions list, config link, 2nd deploy #7367
• E2E: automatic teardown polish #7358
• E2E: improve CLI log format #7364
• E2E: utility - cleanup all #7357
• E2E: utility - cleanup apps #7356 👈 (View in Graphite)
• E2E: utility - cleanup stores #7355
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.