Skip to content

e2e@0.21.0

Choose a tag to compare

@github-actions github-actions released this 14 Oct 17:27
· 62 commits to main since this release
37e9469

Minor Changes

  • #1132 c82a03a Thanks @okwasniewski! - app.command.inheritEnv passes host variables to the app by name or trailing-* prefix (['DATABASE_URL', 'STRIPE_*']), copying each one only when it is set, so a host can hand the app per-run values without listing them in the config or spreading process.env. command.env wins over an inherited value, and inherited values are redacted from a quoted startup failure like command.env. A bare * and a * anywhere but the end are INVALID_CONFIG.

  • #1134 8497c06 Thanks @okwasniewski! - E2E_CACHE_DIR and e2e run --cache-dir <dir> (also on e2e cache) replace cache.dir, so checkouts and worktrees of one project can share a replay cache without editing the config. The flag wins over the variable, both resolve from the project root with ~ or a leading ~/ as the home directory, and the cache mode stays as configured. With a custom cache.store, --cache-dir is an error and E2E_CACHE_DIR is ignored.

  • #1137 27a6ed0 Thanks @okwasniewski! - e2e feedback now asks for reports the e2e team can act on. A --type bug report needs --repro, steps that reproduce it every time. Without it, the command sends nothing and prints a prefilled GitHub issue instead, so the team can ask you for a reproduction there. When the registry confirms the installed e2e is older than the latest release, the report exits 2 and sends nothing, since the problem may already be fixed; upgrade and check again, or pass --allow-outdated. The new --error-code flag takes the e2e error code involved, and --task, --expected, --actual, and --approach now take at most 1500 characters.

  • #1135 78655e1 Thanks @okwasniewski! - e2e init takes a flag per wizard question: --engine, --gateway, --endpoint, --skills, --mcp, and --install/--no-install. A flag skips its prompt; with --yes, an omitted flag keeps its default. Without a terminal, pass --yes. --endpoint implies --gateway openai-compatible. --skills and --mcp take comma-separated locations or none. An unknown value exits 2 before any write and lists the valid ones.

    npx e2e init --yes --gateway openrouter --skills .agents/skills --mcp .mcp.json --install
  • #1097 a869c06 Thanks @okwasniewski! - textContent(), inputValue(), getAttribute(), isEnabled(), isDisabled(), isChecked(), and boundingBox() now wait for their one match until the action timeout, as actions and Playwright's reads do, instead of failing at once with LOCATOR_NOT_FOUND when the node is still rendering. isVisible(), isHidden(), count(), all(), and allTextContents() still answer from the current screen. A read of a node that never appears fails after the action timeout.

  • #1131 babae32 Thanks @okwasniewski! - e2e run --max-duration <ms> and the config key maxDuration stop a run once it has spent that long running tests: nothing more is dispatched, the running tests end as interrupted, the rest are skipped with the new cause time-limit, and the run records a RUN_TIMEOUT error and exits 1. --last-failed runs the interrupted and skipped tests next. The report schema's skip causes gain time-limit.

  • #980 3a53fbd Thanks @bheemreddy181! - Add openHost<Fixtures>() to e2e/runner for other test runners. A worker-lifetime host prepares the target, starts app processes, and initializes its engine once. host.openSession({ name, timeout, signal, agent, onStep }) opens a fresh test-lifetime attempt with typed fixtures, its own deadline, steps, and artifacts. Only one hosted attempt can be opening, live, or closing per process; a host can open sessions sequentially.

    A session's timeout is enforced: when it passes, every operation still running fails with TEST_TIMEOUT, the session is timed out, and close() reports it. A launch that fails or is cancelled leaves the host usable for the next session; only a session whose engine cleanup failed closes the host.

    Closing a session writes a redacted trace.md under a safe, unique results/host/ directory, available through the absolute tracePath for attachments. close() returns soft expectation and cleanup failures; async disposal throws an AggregateError when failures exist. Closing the host cancels any pending or active session, then disposes and releases its worker resources once. Both close methods are idempotent. config accepts a path or an E2EConfig value. The one-shot MCP attempt stays internal.

Patch Changes

  • #1130 2536ed3 Thanks @kacperkapusciak! - Update dependencies: the web engine drives Playwright 1.64.0 (and the Chromium it installs), and smol machines boot the matching mcr.microsoft.com/playwright:v1.64.0-noble image. The runner moves to oxc-parser, oxc-transform, and @oxc-project/runtime 0.153.0, @modelcontextprotocol/server 2.3.1, and @ai-sdk/provider 4.0.26.

  • #1058 a91b6fb Thanks @HEOJUNFO! - An end-mismatch cache detail now says which part of the end check failed: recorded nodes that did not show (missing alert "Acme Notes"), nodes that should have gone but are still there (still showing ...), alerts the recording never saw (unrecorded alert ...), or an outcome that already showed before the replay acted. It used to print the first two recorded nodes, whether or not they were on screen.

  • #1093 0feaefb Thanks @okwasniewski! - The list reporter badges an e2e explore --target <name> run with the explored target rather than the first configured one.

  • #1092 6b1e524 Thanks @okwasniewski! - A judgment (agent.assert, agent.waitFor, agent.extract) on a screen denser than the observation budget now cuts the screen by its instruction's measured size, so a long instruction, or e2e explore's planner once its record of earlier steps has grown, no longer fails the adapter's pre-flight with STEP_BUDGET_EXHAUSTED: model input upper bound N exceeds the agent's maxInputTokens M and aborts the run before any request is sent.

  • #1111 c4d8706 Thanks @kacperkapusciak! - A TypeScript import of a directory through a package subpath or a # import now loads the directory's main or index.ts, the way TypeScript resolves it. A workspace package without exports imported as @acme/db/enums, an exports entry naming a directory, and a # import mapped to one used to fail collection with ERR_UNSUPPORTED_DIR_IMPORT. A relative directory import already worked. Under Yarn PnP, a package subpath that names a directory still fails: PnP's resolver does not say which directory it found.

  • #1107 dbdec70 Thanks @KrzysztofMoch! - Make the bundled agent skill work alongside the shared Codex and Claude plugin, document project-local Codex MCP setup, register the MCP server with Claude Code at project scope in the CLI hints, and ask before sending feedback.

  • #1103 292cd05 Thanks @hasandenizuk! - MCP locate no longer sends a status, alert, or listitem name that matches nothing back to the observation as an accessible name. It says that observe shows those roles by their text and names the getByRole(role).filter({ hasText }) and getByText calls that match that text. A query that already has exact: false is no longer told to loosen the match with exact: false.

  • #1098 a5bb4db Thanks @TMisiukiewicz! - e2e login prints the sign-in URL on a line of its own instead of inside a box, so copying it no longer picks up the box's borders.

  • #1021 33cb8d3 Thanks @Yi-111-a! - e2e models opencode-console lists models through OPENCODE_API_KEY when it is set, the key opencodeConsole() uses in place of the stored login. e2e models with no provider includes OpenCode Console when only the key is set, and with no credential the error names both npx e2e login opencode-console and OPENCODE_API_KEY.

  • #1018 dec61af Thanks @fecolinhares! - opencodeConsole() asks for an explicit output cap on the models the workspace config serves over Anthropic. Those ids are unknown to the Anthropic SDK, which limits a call that sets no maxOutputTokens to 4096 tokens and only warns: a long answer ended with finishReason: 'length' and nothing in the result said why. The provider now asks for 8192, the cap the runner already allows a judgment, and a caller's own maxOutputTokens is left untouched.

  • #1073 14b77d3 Thanks @thymikee! - The mobile engine now bounds each input (a tap or other node action, a point action, typing, a key, back, a keyboard dismissal) by its budget (actionTimeout, or the call's timeout). An input that outlived it used to hold the step until the test timeout. Now an input sent that does not answer in time fails with ACTION_MAY_HAVE_COMMITTED, which the runner never repeats, as on the web engine; one sent with almost no budget left still gets 250 ms to answer. An input the budget ran out before, while its target was still settling, is not sent and fails with OPERATION_TIMEOUT. A long press or timed swipe that holds at least as long as the budget left is refused before it is sent, naming the hold and the time left. Reads (observe, locate, the relocation before a tap), launches, artifacts, and device.* fixture calls are unchanged: a locator poll still lets its last read finish. The OperationContext.signal docs in e2e/engine now say what it does: it aborts with the attempt or the running hook, not at timeoutMs.

  • #1104 762f7de Thanks @DeryFerd! - The bundled agent skill's limrun note on iOS keyboard actions no longer names an agent-device version, which had gone stale.

  • #1108 b965c10 Thanks @kacperkapusciak! - The bundled agent skill and the docs now define an atomic agent.act goal: one outcome that one check confirms. Core concepts shows a bad instruction that joins three goals next to the good version, three act calls each followed by its check, and the examples that joined two checkable outcomes in one act are split.

  • #1096 37fee75 Thanks @okwasniewski! - The run telemetry event now counts agent steps by entry point (steps_agent_act, steps_agent_assert, steps_agent_extract, steps_agent_waitfor) and how many executed tests ran any agent step and how many of those ran agent.act (tests_with_agent_steps, tests_with_act), so the share of tests that are deterministic end to end can be read off a run.