Skip to content

fix(e2e): stop paying opencode's per-directory plugin install ~40 times a run - #2270

Merged
Soph merged 5 commits into
mainfrom
soph/opencode-e2e-diagnostics
Sep 4, 2026
Merged

fix(e2e): stop paying opencode's per-directory plugin install ~40 times a run#2270
Soph merged 5 commits into
mainfrom
soph/opencode-e2e-diagnostics

Conversation

@Soph

@Soph Soph commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

https://entire.io/gh/entireio/cli/trails/1238

What

The e2e-tests (opencode) leg went red on main from 2026-09-03 22:10 UTC and stayed red. This makes it green again: run 33867981097all 56 tests passed, Run E2E Tests 469s (it was 1266s and failing).

Why it was failing

Every failure was agent failed: signal: killed at the 120s prompt budget, with zero assertion failures and completely empty stdout and stderr. Nothing in the artifacts named a cause, because opencode prints nothing while it stalls.

It was not a gradual degradation. The single-process bootstrap warmup was 9–15s for all 32 preceding runs, then 40s+ on every run after. Not ours, and not opencode's code either: same opencode version (1.18.27) on both sides of the boundary, same runner image, only trail-routing commits in our repo, the claude-code leg sharing the same API key got faster, our own hook perf was better in the red runs, and opencode's install path has one commit since Aug 25 (on 08-27).

What changed is that a cost which was always pathological stopped fitting in the budgets.

The mechanism

From opencode's source at 1.18.27: ConfigPaths.directories returns the global config dir plus every existing .opencode directory walking up from cwd. config.ts:452 forks a background @opencode-ai/plugin install for each of them. plugin/index.ts:184if (plugins.length) yield* config.waitForDependencies() — then blocks on all of them, and waitForDependencies joins the fibers with no timeout and logs nothing.

So a repo pays a 27-package, ~61MB install once it has both a project-local .opencode directory and a plugin configured anywhere, global or project. entire enable supplies both by writing .opencode/plugins/entire.ts. Every test gets a fresh repo, so we were paying it ~40 times in parallel.

It is also much worse than an npm install sounds: npm.ts:81 runs @npmcli/arborist's reify() in-process rather than the npm CLI. Measured on one machine, identical 27-package result: arborist 5m00.9s at 1% CPU, npm install 3s.

What this does

  1. Collect opencode's logs into the artifact tree. This is what made the diagnosis possible at all, and it is the piece worth keeping regardless of the rest.
  2. Pre-build the dependency tree once with the npm CLI and seed it into each test repo through a new RepoSeeder hook on SetupRepo. node_modules is symlinked, not copied — 61MB across thousands of small files times ~40 live repos would cost about what it saves. The seed also writes opencode's own .gitignore, which names its own file, so nothing planted is visible to git; these repos are the subject of checkpoint assertions.
  3. Raise the warmup budget and report each attempt's duration, so the one serial, uncontended measurement of opencode startup is visible in the CI log rather than only in a downloaded artifact.

Every step degrades to today's behaviour instead of failing: no npm, an unparseable version, or a platform without unprivileged symlinks all leave the repo alone and let opencode install for itself.

Verification

Don't read the pass count alone — parse the newly-collected opencode-logs/opencode.log for the gap between the last loading path= line and the next event. That is the install window.

broken fixed
median gap 23.5s 0.15s
p90 57.5s 0.24s
killed inside it 31 of 53 6 of 60

The 6 remaining are processes ended by test teardown, not stalls — with a 0.35s max gap there is nothing left to stall on.

Worth knowing

One intermediate revision here looked fixed and was not: Bootstrap runs in go run ./e2e/bootstrap and SetupRepo in go test ./e2e/tests, two processes, so handing the tree between them via a package variable seeded nothing. Because seeding degrades quietly, the bootstrap log reported success while 31 of 53 processes still stalled. The tree is now resolved per-process, and TestOpenCodeSeedRepoPlantsDeps runs SeedRepo from a process that never called Bootstrap — the shape of that bug.

Not in scope

The same install hits real users under the same two conditions. Moving Entire's plugin to ~/.config/opencode/plugins/ would make it a one-time global cost instead of per-repo, at the price of loading in every project including ones where Entire was never enabled — a design call, not a refactor. Upstream: anomalyco/opencode#47212.

🤖 Generated with Claude Code


Note

Medium Risk
Touches only E2E/CI paths but adds cross-process caching, npm installs, and symlink semantics where quiet fallback could mask regressions.

Overview
Fixes opencode E2E timeouts caused by ~40 parallel, silent @opencode-ai/plugin installs (~61MB each) after entire enable adds a project plugin.

Adds a RepoSeeder hook so agents can plant files post-entire enable; opencode SeedRepo now writes opencode.json, copies pinned package.json / lockfile into .opencode, symlinks a shared node_modules tree built once per process via npm install (version-keyed under os.TempDir()), and mirrors opencode’s .gitignore so seeded blobs don’t affect checkpoint/git assertions. Bootstrap warms up in a seeded scratch dir through RunPrompt with longer first-attempt budgets and per-attempt timing logs.

CI copies ~/.local/share/opencode/log into artifacts on the opencode matrix leg. TestOpenCodeSeedRepoPlantsDeps asserts seeding works from the test process without relying on bootstrap-only state.

OpenCode-specific setup was removed from SetupRepo in favor of the seeder interface; failures to pre-build or link deps degrade to opencode’s own install rather than failing setup.

Reviewed by Cursor Bugbot for commit b055dcb. Configure here.

Soph and others added 4 commits September 4, 2026 12:22
The opencode leg went red on 2026-09-03 and every failure looked the
same: `agent failed: signal: killed` at the 120s prompt budget, with
empty stdout and empty stderr. opencode prints nothing while it stalls
during startup, so the artifacts named no cause — the leg was
undiagnosable after the fact, and the elimination had to be done from
timing alone (the CLI's own hooks were fine, and faster than in the last
green run, so the added cost is inside opencode's startup).

Fold opencode's own log directory into the artifact tree so the next
occurrence carries the stalled call with it. Best-effort: a leg that
never got as far as writing logs must not fail the step.

Also raise the warmup budget from 30s to 90s and report each attempt's
duration. The warmup exists to pay opencode's first-run costs once and
serially before ~40 tests start in parallel; at 30s it was being killed
part-way through on every run since the step change, leaving that work
half-done for every test that followed. Reporting the duration makes the
one serial, uncontended measurement of opencode startup visible in the
CI log instead of only in a downloaded artifact.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01M1NZ1KYM0K1S73B1EHB9Z9KK
…m per repo

Whenever a project directory contains a local plugin file — which `entire
enable` always writes — opencode generates `.opencode/package.json`,
pinning `@opencode-ai/plugin` to its own binary version, and installs it
(27 packages, ~61MB) before it will finish bootstrapping. The install
runs inside a phase that logs nothing and has no timeout, so a slow one
is indistinguishable from a hang: the process sits between "loading
path=..." and "all LSPs are disabled" and produces no output at all.

Every test gets a fresh repo, so that install was being paid ~40 times
per run, in parallel. Since 2026-09-03 it stopped fitting in the budgets:
in run 33862929010, 27 of 53 opencode processes died inside that window.
Measured here, an unseeded directory takes 5m00.9s at 1% CPU to complete
an install that takes 3s when npm is invoked directly.

Build the tree once in Bootstrap, keyed by opencode version (the pin
follows the binary, so an upgrade invalidates it), and plant it in each
repo from a new RepoSeeder hook on SetupRepo. node_modules is symlinked
rather than copied: at ~61MB across thousands of small files and ~40
repos live at once, copying would cost about what it saves. The seed also
writes the same `.gitignore` opencode writes, which names its own file,
so nothing we plant is visible to git — these repos are the subject of
checkpoint assertions.

Measured with a seeded directory: bootstrap reaches the model in 839ms.

Every step degrades to today's behaviour rather than failing: no npm, an
unreadable version, or a platform without unprivileged symlinks all leave
the repo alone and let opencode install for itself.

Note this is not a test-only workaround. Every user running opencode pays
the same install on first launch in each repo, and again after each
opencode upgrade, with a blank screen and no explanation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01M1P10839YG3R5AF274XYE6TT
The pre-seed landed but did nothing for the tests. Bootstrap runs in
`go run ./e2e/bootstrap` and SetupRepo in `go test ./e2e/tests` — two
processes — so the package variable Bootstrap set was always empty where
SeedRepo read it. Seeding degrades quietly by design, so the failure
looked like success: run 33865617639 reported deps built in 4.9s and a
warmup that finally passed on the first attempt, while 31 of 53 opencode
processes still died inside the install window.

Resolve the tree through a sync.OnceValues instead, so either process
finds it on disk or builds it. It lives under os.TempDir() rather than
os.UserCacheDir() for two reasons: the e2e TestMain points XDG_CACHE_HOME
at the artifact directory, so a tree built there would both miss the one
Bootstrap built and be uploaded as a ~61MB CI artifact.

The install is staged and renamed into place, so a reader never sees a
half-installed tree and a process that loses the race adopts the winner's.

The new test runs SeedRepo from a process that never called Bootstrap,
which is the shape of the bug — the previous version passed every unit
test and every local check while seeding nothing.

Also narrows the claim in Bootstrap's comment to what the opencode source
actually does: it forks the install for every existing config directory
and blocks on all of them whenever any plugin is configured, so a repo
pays only once it has both a project-local .opencode directory and a
plugin configured anywhere. `entire enable` supplies both.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01M1P2Q273Z8W236F5RE4GV00B
Cleanup pass over the preceding two commits; no intended behaviour change
beyond the warmup budget and the warmup dir now getting the same config
the test repos get.

- The warmup went through a hand-rolled exec that had become a copy of
  RunPrompt once it needed cmd.Dir and openCodePromptEnv. Call RunPrompt.
  That env helper exists because opencode resolves its project root from
  PWD, which cmd.Dir does not set, so a divergence there warms the wrong
  directory silently — one call site is the point.
- Retry at 30s rather than 90s. Only the first attempt can be paying
  opencode's real first-run costs, so 3x90s was over four minutes of
  blocked CI on the exact path the warmup exists to survive.
- SetupRepo wrote opencode.json from an agent.Name() switch ten lines
  above the new RepoSeeder call, leaving two homes for one agent's
  per-repo setup. Moved into SeedRepo, ahead of the dependency-tree
  resolution so a repo still gets a usable config when the pre-build
  failed.
- Use the package's build-tagged linkFile instead of a raw os.Symlink
  plus a comment restating what its Windows variant already documents.
- Drop Bootstrap's deps if/else: buildPluginDeps reports the build and
  SeedRepo reports a failure, so it only added a third message about one
  event.

Skipped from the review, with reasons: memoizing package.json and the
lockfile alongside the path (measured under a millisecond across all ~40
repos, and it trades a plain path for a struct); pruning the collected
opencode log dir (estimated at 15-50MB, measured at 464KB); moving log
collection from the workflow into TestMain and documenting the install in
the agent guide (both real, both outside this diff).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01M1P4SARPGWY5DMEJV74TNP4T
@Soph
Soph requested a review from a team as a code owner September 4, 2026 12:03
Copilot AI lite review requested due to automatic review settings September 4, 2026 12:03
jdx
jdx previously approved these changes Sep 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new seeding test currently assumes symlinked node_modules unconditionally and should be excluded from Windows builds to avoid expected failures.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR fixes recurring e2e-tests (opencode) timeouts by avoiding Opencode’s per-repo plugin dependency install during startup, while also improving diagnosability and making warmup behavior more observable.

Changes:

  • Introduces an agents.RepoSeeder hook and updates repo setup to call it after entire enable, allowing agent-specific pre-run seeding.
  • Implements Opencode repo seeding + once-per-process dependency prebuild (via npm install) and updates Opencode warmup budgeting/logging.
  • Collects Opencode’s own logs into CI artifacts for post-failure diagnosis.
File summaries
File Description
e2e/testutil/repo.go Calls an optional RepoSeeder after enabling Entire, so agents can plant required files once per fresh repo.
e2e/agents/agent.go Adds the RepoSeeder interface and documents its best-effort contract.
e2e/agents/opencode.go Adds Opencode dependency prebuild + repo seeding and improves warmup budgets/timing output.
e2e/agents/opencode_seed_test.go Adds a regression test to ensure seeding works from the test process (not relying on bootstrap-only state).
.github/workflows/e2e.yml Copies Opencode’s log directory into the artifact tree on the opencode CI leg.
Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread e2e/agents/opencode_seed_test.go
…n Windows

Two review findings.

sync.OnceValues cached a failure exactly as permanently as a success, so
one transient npm error on the very first call would disable seeding for
every repo in the run — reinstating the concurrent-install storm this
change exists to prevent, for the whole run instead of one attempt, with
only a stderr line as evidence. A success is still cached for the life of
the process; a failure is now retried, capped at two attempts total so
the opposite failure (~40 installs when npm is simply absent) stays out
of reach. The lock is held across the build either way, so attempts are
serial.

The seed test asserted a symlinked node_modules unconditionally, which
cannot hold on Windows: linkFile copies there, a copy cannot be a 61MB
directory, and SeedRepo takes its documented degradation path. Rather
than excluding the file from Windows builds — which would also drop the
platform-independent assertions, including the cross-process resolution
the test exists to pin — it now asserts the degradation on Windows and
the symlink elsewhere. Not a live CI failure: the Windows job runs only
./cmd/entire/cli/... filtered to (Windows|MSYS), so this was reachable
from a local Windows run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Entire-Checkpoint: 01M1P74VTN4TAQXTCD7K87HJ1Q
@Soph
Soph merged commit 6219a74 into main Sep 4, 2026
19 checks passed
@Soph
Soph deleted the soph/opencode-e2e-diagnostics branch September 4, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants