Skip to content

e2e@0.19.0

Choose a tag to compare

@github-actions github-actions released this 09 Oct 13:54
· 266 commits to main since this release
cae1028

Minor Changes

  • #947 f3aa4c8 Thanks @kblok! - locator.dblclick() is an alias of doubleTap(), as click() is of tap(), so a test ported from Playwright or Cypress keeps its double clicks.

  • #869 1b92a6d Thanks @okwasniewski! - trace keeps its modes but now writes e2e's own trace instead of a Playwright zip: a markdown page per test, on every engine, and the terminal names it under each failed test (❯ trace .e2e/results/<test>/trace.md). The default is retain-on-failure, locally and in CI. Each step on the page shows what it did: the node an action landed on, the values an assertion or waitFor read while it waited, the replay cache's decision (where a replay stopped and why, what became of the recording, the entry file), what the app logged meanwhile (console output, uncaught exceptions, failed requests, and 4xx and 5xx responses, also gathered in one App log section with each line's step), where the page went (a navigation, a new tab, a frame), and how the screen changed since the step before. The page also names the browser and user agent, or the device, the test ran on. The markdown reporter's summary.md names each failure's trace page when one was kept. report.json is still written for scripts, but the list summary no longer prints its path (--debug does), and prints Report (not written) only when it is missing. Everything a test left now sits in one directory, <output>/results/<test>/ (.e2e by default): its trace.md (which shows the failure screenshot inline and links each file) and an attempt-<n>/ per attempt, numbered from 1, with the screen at failure as screen-at-failure.txt. It replaces .e2e/artifacts/<target>/<test id>/<agent>/attempt-<n>/; a --last-failed rerun writes under each test's rerun-<n>/. Point CI uploads at .e2e/results, and add .e2e/results/ to .gitignore (or rerun e2e init). A stale recording made from this version on names what changed in its key under --strict-cache, such as the agent's context. For engine authors: startTrace and stopTrace are no longer called, trace leaves artifactCapabilities, EngineAttemptContext.appLog takes what the app logs, navigation where it went on its own, screen takes a screen for the trace (one a locate matched against, or one read right after an action, bounded), and environment takes what the attempt runs on. report-1 drops the trace artifact kind and the not-required redaction (StoredArtifact.kind and redaction narrow the same way), drops TRACE_WITHHELD, and adds attempt.appLog (each line with its step), navigation events, failure.viewport and failure.nodes, step.phase, step.screen, attempt.environment, and step.cache.entry, detail, and write (report only; result.cache from agent.act() gains entry and detail). FinishedRun.traces maps results to their pages.

  • #965 b92658b Thanks @okwasniewski! - A custom StepExecutor that declares vision: true receives agent.assert(..., { vision }) as ctx.step.vision instead of UNSUPPORTED_CAPABILITY.

  • #978 2f60b54 Thanks @mvanhorn! - list from e2e/runner reports the tests a config, file selection, and title grep would select, in process, without starting an app, an engine, or a worker. The result includes pairs a filter removed, positional arguments that matched no file, and the selected target names. e2e list --reporter json renames skipReason to reason and adds each pair's identity: id, source, session, sessions, serialId, and agent, beside the fields it already printed.

  • #924 d3d9081 Thanks @okwasniewski! - Breaking: a failed expect.soft before test.skip(...) now fails the test, as in Playwright. Before, the test was reported skipped and the run exited 0 unless failOnSkippedFailure was set. The attempt is failed with the soft failures as its error, keeps the skip reason in its skip field, and retries like any failure. In report.json, an attempt or serial member may now carry skip with a failed status. The CLI prints the reason under the failure, and the trace page (trace.md) names it. failOnSkippedFailure now covers only a retry that skips after an earlier failed attempt.

  • #977 5137778 Thanks @okwasniewski! - Add toHaveScreenshot for visual comparison on every engine that captures pixels: await expect(screen).toHaveScreenshot('home.png') compares the screen and expect(locator).toHaveScreenshot() one element against a PNG stored beside the test file, one per target and operating system. A missing one fails the test and is written beside it, or in CI attached to the results instead; --update-snapshots (-u) writes missing and different ones and passes. Options: threshold, maxDiffPixels, maxDiffPixelRatio, mask, maskColor, timeout. A mismatch attaches the expected, actual, and diff images to the step. On iOS and Android the status bar is held in a fixed state for these screenshots.

Patch Changes

  • #893 7eb03ca Thanks @rudycelekli! - Stop a fixture setup that finishes after its attempt was abandoned from starting later fixtures, hooks, or the test body. The late fixture still releases its own teardown.

  • #921 f2cf2ad Thanks @okwasniewski! - Agent steps on Meta's Muse Spark models, through AI Gateway or OpenCode Console, no longer fail with MODEL_PROVIDER_FAILED: only auto is supported for tool_choice. The refusal now reads as one, so the step retries with auto and the tool-calls-only rule, as it does for Anthropic and DeepSeek.

  • #929 e77c17a Thanks @okwasniewski! - The replay cache key no longer carries the engine's version, so upgrading @e2e-dev/web or @e2e-dev/mobile keeps every committed recording. A recording re-finds its nodes at replay, which is where a node an engine release resolves differently is caught. Entries recorded before this release sit under the old keys: a read-write run records them again, and --strict-cache names them as REPLAY_STALE until it does.

  • #935 3c37d98 Thanks @rudycelekli! - Cancel the ChatGPT authorization-code token exchange when sign-in is interrupted.

  • #901 67b11b5 Thanks @rudycelekli! - Skip MCP tool execution when the client cancels a queued call or cancels during argument validation.

  • #936 5cdd3c6 Thanks @rudycelekli! - Cancel in-flight app and service readiness HTTP probes when the run is interrupted.

  • #899 d5a4eab Thanks @rudycelekli! - Cancel the initial OAuth device-code request promptly when login is interrupted, without showing or saving a cancelled login.

  • #918 9fb70b9 Thanks @okwasniewski! - A credential whose username is not a string, or with a key other than username and password, fails the config load with INVALID_CONFIG naming the field, instead of a raw "value of type function is not JSON-safe" error. Only password may be a function. A credential with no username in the config or E2E_USER_<NAME>_USERNAME is INVALID_CONFIG too.

  • #907 fef1246 Thanks @rudycelekli! - Keep direct locator reads within their original action timeout when a stale reference requires resolution again.

  • #887 4132715 Thanks @DeryFerd! - e2e init no longer writes an MCP config through a symlink: a .mcp.json or .cursor/mcp.json reached through a link, or under a linked directory, is skipped with a warning naming the link and its target, the way a linked skill directory already was. Before, the merge read and overwrote the file the link pointed at, which could be a dotfiles repo or a shared config outside the project.

  • #919 b6efcf7 Thanks @okwasniewski! - store: false and a prompt cache key now go only to OpenAI and Azure OpenAI Responses models and to openai/* models on the gateway. An OpenAI-compatible provider named openai no longer fails with Unknown parameter: promptCacheKey.

  • #920 745b353 Thanks @okwasniewski! - A polling assertion takes its last read at its timeout instead of up to 100 ms after it, so it no longer passes on a state that arrived after the timeout.

  • #922 c10dc00 Thanks @okwasniewski! - When the provider package can't read a provider's error body, MODEL_PROVIDER_FAILED no longer shows an empty message. One example is the ChatGPT backend's HTTP 400 {"detail": "Unsupported service_tier: fast"}. The message and the --ai-trace entry now show the HTTP status and the body, with secrets redacted and the body cut at 1 KB. An agent.act step that used up its transport retries now names the last attempt's failure, as judgments already did.

  • #905 ea0681a Thanks @rudycelekli! - Reject read-only environment credentials before starting a provider login that cannot be saved.

  • #923 99d5ab5 Thanks @okwasniewski! - --strict-cache (and cache.strict) never writes or deletes a cache entry, even in read-write mode, and retries replay like the first attempt. Before, a retry skipped replay, called the model, and overwrote the committed entry, and a failed attempt could evict it.

  • #984 bc453bd Thanks @ASVLCII! - Align e2e cache ls columns by displayed width for test file names containing wide characters, combining marks, or emoji.

  • #881 cae1c65 Thanks @fecolinhares! - A recorded step the cache hands to the agent no longer accumulates the replay wait or the agent's turn into the next replay's end wait. The wait is measured from the agent's verdict if it settled without acting, or from its last action if it acted.

  • #927 0ddea76 Thanks @rudycelekli! - Match a Unicode code point with the ? test glob wildcard, so a test file named with one emoji is selected by a single ?.

  • #998 841ea3f Thanks @okwasniewski! - A recorded tap on a row of a windowed list no longer depends on which row the list rendered first. The container key of a row was its list's first text, a scroll position on Android (Row 0512 in Row 0500), so the replay found the row and still handed off whenever the window started one row later.