Fix flaky E2E tests with API-based setup and data isolation

[choo121600]

Summary

E2E tests have been intermittently failing in CI, causing unreliable builds and frequent reruns, due to two root causes:

• UI-based data setup in beforeAll
  • Tests created seed data by opening a browser and clicking through the UI. This made setup fragile: dialog animations, rendering delays, and element timing issues could cause the setup itself to fail before any assertions ran.
• Insufficient data isolation between parallel workers
  • Test identifiers were generated with Date.now(), which can collide when multiple Playwright workers start at the same time. Leftover data from previous runs could also interfere with assertions that assumed exact row counts.

This PR makes E2E tests deterministic and reliable by removing UI-driven setup and enforcing strict data isolation.

Change

New shared utilities (tests/e2e/utils/test-helpers.ts)

Replaces fragile UI-based setup with reliable API-driven helpers:

• uniqueRunId(prefix)— UUID-based ID generation to prevent cross-worker collisions.
• waitForDagReady() — Polls GET /api/v2/dags/{id} until the Dag is parsed and available.
  apiTriggerDagRun() / apiCreateDagRun() / apiSetDagRunState()* — Create and manipulate Dag runs via the API with 409-conflict handling for parallel safety.
• waitForDagRunStatus() / waitForTaskInstanceState() — Poll until a Dag run or task instance reaches the expected state.
• apiRespondToHITL() / setupHITLFlowViaAPI() — Drive the full HITL operator flow via API, removing the need for browser-based setup.
• apiCreateVariable() / apiDeleteVariable() — Variable CRUD.
• apiCreateBackfill() / apiCancelBackfill() — Backfill lifecycle management with automatic retry on 409.
• waitForTableLoad() / waitForStableRowCount() — DOM stability helpers for reliable table assertions.

Custom Playwright fixtures (tests/e2e/fixtures.ts)

Standardizes test setup and removes boilerplate across all specs:

• All Page Object Models are now provided as Playwright test fixtures, removing manual new PageObject(page) usage in every spec.
• authenticatedRequest (worker-scoped) provides an APIRequestContext with stored auth state, enabling API calls in beforeAll/afterAll without creating a browser context.

Spec and Page Object updates (22 specs, 18 page objects)

Every spec file follows the same migration pattern, ensuring consistency:

• import { test, expect } from "@playwright/test" --> import { test, expect } from "tests/e2e/fixtures"
• beforeAll setup: browser.newContext() + UI interactions --> authenticatedRequest + API helper calls
• Date.now() identifiers --> uniqueRunId() for collision-free IDs
• afterAll cleanup: tracks created resources and deletes via API(404 treated as success for idempotency)

After

• Setup failures eliminated — Data creation no longer depends on UI rendering, dialog animations, or element timing. API calls either succeed or return a well-defined status code.
• Parallel-safe by design — UUID-based identifiers (uniqueRunId()) guarantee zero collisions between concurrent Playwright workers.
• 409 conflict handling — All API helpers treat 409 Conflict as acceptable (resource already exists), making tests idempotent across retries and re-runs.
• Faster simple setup — Straightforward data seeding (variables, connections) completes in seconds rather than tens of seconds, since it no longer requires browser rendering. Complex flows like HITL still require extended timeouts but are more reliable since they bypass UI interaction entirely.
• No extra browser contexts — beforeAll no longer opens a separate browser.newContext() just for seeding data, reducing memory pressure and startup overhead per worker.

related: #63036

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  claude

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[bbovenzi]

Nice! Could we possibly break this up into smaller PRs?

It is a lot of changes to review at once.

[choo121600]

@bbovenzi Thanks!
UI E2E tests have been pretty unstable, and this PR helped validate some improvements in stability. (Finally!)

I agree it's quite large — it currently mixes structural changes with fixes for flaky tests.
Given the current test structure, it was also difficult to identify root causes, so I worked in a single branch.

I’ll split it into smaller PRs soon 😉