Skip to content

e2e@0.23.1

Choose a tag to compare

@github-actions github-actions released this 13 Nov 13:26
· 53 commits to main since this release
dd3aa19

Patch Changes

  • #1710 0920ef5 Thanks @kdy1! - The agent tells the model when an action opened a tab in the background (The app opened a new tab (tab 2 of 2). The screen still shows tab 1; use switch_tab to see it.), names the tab in every screen header while more than one is open (tab 1 of 2), and gets switch_tab and close_tab verbs on engines that declare the new session.switchTab and session.closeTab hooks. EngineSnapshot.tabs reports the open tabs. Action tools

  • #1600 f5c9099 Thanks @okwasniewski! - The built-in agent can try an upload when the step names no file. upload takes sample:photo.png, sample:photo.jpg, sample:document.pdf, sample:notes.txt, sample:table.csv, and sample:empty.txt, small files e2e ships, and the tool lists them. An e2e mcp upload of a sample prints no test line, since a test needs a project file.

  • #1541 11dea90 Thanks @samanyudas! - Fail e2e run when an app command exits unexpectedly after readiness.
    Report its exit code or signal and redacted log tail.
    Interrupt active tests and skip queued tests only for targets that depend on that command.
    Ignore command exits after those targets finish their work.
    An exit with code 0 also fails the run, so a command that starts the app in the background and then exits, such as docker compose up -d --wait, now fails.
    Keep the app in the foreground instead, for example docker compose up without -d.

  • #1577 9289ff4 Thanks @DeryFerd! - A startup failure now quotes the first lines of a long command.log as well as the last, with [startup output omitted] between them, so the error a service printed before it kept logging survives in the APP_UNREACHABLE message.

  • #1568 1144df4 Thanks @bheemreddy-samsara! - Add opt-in scrolling exploration to agent.assert with bounded action and model-call budgets. Enforce read-only assertion tools for built-in and custom executors. ACP assertions now deny hover; mutating actions from custom executors fail with POLICY_DENIED during any assertion. createVerdictTool(kind) drops ASSERTION_INCONCLUSIVE from an act verdict, and createToolLoopExecutor({ inconclusiveOnOverflow: true }) fails a context overflow as ASSERTION_INCONCLUSIVE instead of shortening the history.

  • #1609 34a260c Thanks @okwasniewski! - Bump playwright-core to 1.65.0 and commander to 15.1.0. fill no longer rejects malformed date, time, color, and range values, and actions wait for an element to leave an inert subtree. Smol sandboxes use the Playwright 1.65.0 image.

  • #1666 795f71f Thanks @okwasniewski! - Bump runtime dependencies: @clack/prompts 1.9.0, oxc-parser, oxc-transform, and @oxc-project/runtime 0.158.0, @ai-sdk/provider 4.0.37, and @agentclientprotocol/sdk 2.5.0 (ACP schema v1.30.0).

  • #1616 5847701 Thanks @okwasniewski! - The CLI runs on Bun 1.4.3 or newer without Node.js: bunx --bun e2e run. In a Bun project, e2e init registers the MCP server with bunx. An older Bun gets the version to upgrade to. What differs on Bun.

  • #1586 3c902b0 Thanks @kdy1! - Register the MCP server in Codex project settings during e2e init. Preserve existing Codex settings and comments when updating the server entry.

  • #1730 7e309c1 Thanks @karolkulesza! - The setup guide's link to the command reference no longer returns 404. The e2b and HTML reporter READMEs link to e2e.com instead of e2e.tester.army.

  • #1742 8077542 Thanks @bardrop! - Reject a config timeout above 2147483647 ms with INVALID_CONFIG instead of letting every test time out at once.
    This covers timeout, launchTimeout, actionTimeout, assertionTimeout, cleanupTimeout, app.command.startupTimeout and shutdownTimeout, and agents.<name>.judgmentTimeout.
    A test's or group's own timeout above that limit now fails collection too.

  • #1393 046bd08 Thanks @KrzysztofMoch! - Engines receive the app's origin as EngineAppInfo.origin in prepare and init, on the run's port when app.url asks for a free one (and left out until the run assigns it), so an engine can scope a credential it injects to the exact origin.

  • #1711 17a2e46 Thanks @KrzysztofMoch! - e2e explore --changed-since <ref> explores what a branch changed. The planner reads the commit messages and the committed diff since the merge base of the ref and HEAD, and plans its steps for the screens the change can affect. Only a defect the change can cause is an issue; one elsewhere is a warning. With no goal, the goal is the change. The report records the change as run.explore.change, and explore() on e2e/runner takes changedSince too.

    npx e2e explore --changed-since origin/main
  • #1711 17a2e46 Thanks @KrzysztofMoch! - e2e explore --changed-since can skip a change no user can see, such as docs or tests only. It skips only with no goal given, and only when the planner read the whole change on a clean working tree. The run passes with no step, run.explore.ended reads skipped, and the comment headline reads nothing to explore.

  • #1672 3735887 Thanks @okwasniewski! - Move GitHub repository metadata and CLI links to e2e-inc/e2e after the organization rename. Register the MCP server as io.github.e2e-inc/e2e.

  • #1526 ed9eb0e Thanks @Mohsintl! - The junit reporter links failure screenshots using GitLab CI's [[ATTACHMENT|<path>]] format in <system-out>, so screenshots show directly in pipeline test reports.

  • #1515 ed9e633 Thanks @two-pack! - agent.assert, agent.waitFor, and agent.extract now see the current location, so an instruction about the URL can hold or fail instead of ending inconclusive. agent.waitFor also judges again when only the location changed, except under vision: 'only', which does not see the location.

  • #1717 258ef9b Thanks @KrzysztofMoch! - ENGINE_NOT_INSTALLED from e2e mcp now tells the agent to ask the user to run the install command when it cannot run it itself.

  • #1665 3c9de80 Thanks @KrzysztofMoch! - Open an MCP session in a directory with no config: open_session {engine: "@e2e-dev/web", app: {url}} names the engine to drive and app the app under test. engine is a package name, or the factory call a config would write as {package, export, options}, where an option can be another such call: {package: "@e2e-dev/mobile", options: {platform: "ios", device: {package: "@e2e-dev/limrun", export: "limrun"}}}. Each package is found in the project, beside e2e, or in ~/.e2e/engines; when it is in none, ENGINE_NOT_INSTALLED names the install command. @e2e-dev/web and @e2e-dev/mobile export their factory as the default export for this, and mobile() called without a platform now says which platforms it takes.

  • #1696 de8ae3c Thanks @kacperkapusciak! - e2e mcp tells the agent to read the writing-tests guide before it writes or edits a test, in the server's instructions and in every open_session result.

  • #1208 cc94232 Thanks @piotrpasztor! - A step whose effect or click target is a node identified only by a generated test id (a uuid, hex id, cuid, or timestamp) no longer fails replay on every run.

  • #1587 34ccc5f Thanks @kdy1! - PNG decoding and encoding now run on @napi-rs/image, a native addon with a prebuilt binary per platform, in place of fast-png and pngjs: decoding is 8 to 15 times faster, encoding 5 to 8 times, and a toHaveScreenshot capture 4 to 12 times. Stored screenshots decode to the same pixels as before. A screenshot the runner writes (a new baseline, or one rewritten because the screen changed) has the same pixels in different bytes: a Chromium capture comes out about 15% larger and a device screenshot about 20% smaller. The binary adds about 15 to 21 MB to an install, depending on the platform, and the package publishes none for Linux on ppc64, s390x, or riscv64. e2e loads it when the first screenshot is read or written, so --version and a run that takes none never need it, and a missing binding fails that call with IMAGE_CODEC_UNAVAILABLE; @e2e-dev/mobile and @e2e-dev/decision load it when they are imported. Gray PNGs with 1, 2, or 4 bits per sample now decode, and a 16-bit gray or RGB PNG with a transparent color now compares pixels to it at 16 bits, so a pixel that only rounds to it at 8 bits stays opaque.

  • #1721 565e685 Thanks @KrzysztofMoch! - The bundled e2e skill's running guide now says --output must be a directory inside the project root, not the root itself.

  • #1589 b5e4703 Thanks @kdy1! - A test filter can score the tests it keeps. relevance in its result gives each test a number from 0 to 1, and a run with --changed-since starts the files of the likeliest first in each target, in bands at 75%, 50%, and 25%, so a failure shows sooner and --max-failures stops earlier. Targets go by the band of their best file, ties in config order. Inside a band, a run with more than one worker still starts the file the previous run spent longest in first.

    decisionFilter() now scores every test it keeps: 100% for a test whose file, setup file, or stored screenshots changed, the model's answer for one it judged, and 50% for one it did not judge. When it fails, as without a key, it puts the scores the change alone gives on the error it throws, so the run still warns and runs every test, with the touched ones first. A filter of your own can do the same: an error it throws may carry relevance. decisionFilter() requires e2e 0.23.1 or newer, since an older runner refuses the new relevance key and drops the whole filter.

  • #1431 382b8c4 Thanks @21prnv! - A replay no longer fails its end check on a value that changes every run, such as a clock or a date, when the other side of the step's delta already holds a stable result like a deleted row.

  • #1642 a133807 Thanks @KrzysztofMoch! - A reporter's onRunFinished gets run.redact(text), which masks the config's secret values the way a test filter's redact does, for text it adds from outside the report, such as lines of a test file.

  • #1662 d01bd6d Thanks @Akarshkushwaha! - When --target selects configured targets, the run header and the run-started event's targets name those targets in config order instead of every configured target. Without --target, or with an unknown target ID, they still name every configured target. The report's run.targets still lists every configured target.

  • #1540 93c93f5 Thanks @KrzysztofMoch! - e2e run --step runs the first step, then holds each later step until you press a key, with the keys of --watch: Enter runs it, n runs the rest of the test, a runs the rest of the run, h lists the keys, and q stops the run with exit 130. Add --headed to watch the app between steps. The time a step is held counts toward no timeout, so a test can wait at a prompt past its timeout and maxDuration. Setup tests, retries, teardown steps, and steps inside an expect.poll read run without stopping. --step runs one worker and is refused in CI, without a terminal, with --watch, with --workers above 1, and with the json reporter.

  • #1331 d1b0ff6 Thanks @fghpdf! - Add screen.doubleTapAt(point, options?) to double-tap a viewport point on engines that support the doubleTap pointer action.

  • #1570 34f85c3 Thanks @kdy1! - scroll_to({ text }) is faster: it reads each page once when the swipe visibly landed, instead of waiting for a second look to confirm it. A clock ticking beside a list no longer delays the end of the list: the call fails with LOCATOR_NOT_FOUND after a few swipes, not 800 screens later.

  • #1614 f6aba97 Thanks @okwasniewski! - scroll_to({ text }) keeps paging a list that appends when it reaches the end. An unchanged screen is re-read for a second before it counts as stopped, so that pause is not LOCATOR_NOT_FOUND.

  • #1715 be460ec Thanks @okwasniewski! - Redact project tool results and errors before the model or AI trace recorder reads them, and redact and bound skip reasons before reporting. Avoid repeated scanning of malformed stack frames and trailing slashes in provider API endpoints. Keep ACP forwarding errors out of HTTP responses while preserving the cause in the runner's redacted failure output.

  • #1601 0f158a3 Thanks @okwasniewski! - The bundled e2e skill's bug bash writes each repro test from the finding as agent.act steps and a closing agent.assert, so confirming a bug needs no locators. Each verifier runs its own repros into its own output directory, and the report runs them all once with --tag bugbash. A finding is confirmed when its repro fails on the reported defect, and the skill says how to read a failed act. Charters get about fifteen minutes each, a bash under a time limit keeps running new charters until near the end and reports each bug as its repro confirms it, and an account with a password under 6 characters is called out before the config fails.

  • #1591 937ea15 Thanks @kacperkapusciak! - The bundled e2e skill's GitHub Actions workflow now caches the Playwright browsers between runs.

  • #1664 14dc946 Thanks @Akarshkushwaha! - The bundled e2e skill's running reference now gives e2e doctor --json for JSON output, the flag the command accepts, instead of --reporter json.

  • #1659 00e19d6 Thanks @karolkulesza! - The bundled e2e skill's setup reference shows how to run an iOS app in another language with app.launchArguments, one target per language.

  • #1694 dc01aaa Thanks @kacperkapusciak! - The bundled e2e skill now writes tests with one top-level describe per feature, one behavior per test, and arrange, act, assert order with every assertion at the end, instead of a check after each agent.act.

  • #1627 b0e3012 Thanks @karolkulesza! - The bundled e2e skill now says what timeout: 0 and a negative timeout do on actions, waitFor, matchers, and expect.poll.

  • #1639 b285835 Thanks @kdy1! - e2e run keeps a small <output>/timings.json (schema timings-1) of how long each file took, smoothed over recent runs, and starts the longest files first from it when it runs with more than one worker. A CI cache can restore and save just that file, so a fresh checkout gets the ordering; --timings <file> moves it. A missing or invalid file is no history, and the previous report.json is still the fallback.

  • #1511 d784448 Thanks @Mohsintl! - A replay no longer skips text like Marketing 12 as a date when checking its end state.