AI Agent

AI Agent

An AI agent (Claude or OpenAI) that investigates a failing report, tells you whether it's a real bug, a flaky test, or a bug in the test itself, and — for a test-code bug — opens a pull request with the fix. It never pushes to your repo's default branch or merges anything; a PR is the only way its work reaches you.

Connecting it

Settings → Integrations → AI (owner only, instance-wide — every project shares these keys):

Field	Notes
Anthropic API key	A Console API key (sk-ant-…), from console.anthropic.com — not a Claude.ai subscription login, those aren't interchangeable and are rejected.
Anthropic model	Defaults to claude-sonnet-5 if left blank.
OpenAI API key	An API key (sk-…) from the OpenAI platform.
OpenAI model	Defaults to gpt-5 if left blank.

Both keys are verified against the provider itself before they're saved — a bad, expired, or wrong-type key is rejected immediately with a clear reason, never silently stored. Connect either provider; if both are connected, Claude is used. A blank key field on save means "leave the stored key unchanged," never "clear it" — there's no way to unset a key from the UI once it's saved.

Settings → Integrations → GitHub (owner only) holds a single fine-grained Personal access token, repo scope plus Administration: write if you want Plum to create new repos for you. This is the same token used for connecting a project's repo — the agent's pull requests go through it too.

Below the org-wide keys, Settings → Integrations → AI also has two per-project fields (owner, or an admin on the project), under Agent behaviour:

• Personality & tone — free text, e.g. "Be terse. Ask before deleting a test."
• Coding conventions — free text describing how this repo likes its tests written, e.g. "Use the Page Object Model. Prefer data-testid selectors."

Both are injected into the agent's prompt for every analysis on this project. Without a key connected, the AI Analysis card says so and there's nothing more to do until an owner connects one.

Analyzing a failing report

Open any failing report — an AI Analysis card sits above the report content. Click it, confirm, and the agent:

1. Reads the failing scenario(s), each one's recent pass/fail history, and a screenshot if one was captured.
2. Optionally re-runs the test on the node(s) selected in the run bar, if the history alone isn't conclusive — a re-run takes a node away from the team, so it only does this when it needs to.
3. Settles on exactly one verdict:

Verdict	Meaning
Test code issue	The E2E test itself is wrong (stale selector, bad assertion, missing wait). The only verdict that unlocks Fix it and open a PR.
Real issue	The application under test is genuinely broken.
Flake	Nondeterministic — the test and the app both check out.
Inconclusive	Not enough evidence either way.

The card shows a plain-language headline, a summary, one finding per scenario it looked at (with the evidence it based that on), and a recommendation. Re-analyze starts over from a different angle rather than restating the same conclusion — useful if you disagree with the first pass.

Fixing a test-code bug

Only offered for a Test code issue verdict. Confirm Fix it and open a PR, and the agent:

1. Clones the project's connected GitHub repo into an isolated workspace, on its own branch.
2. Makes the smallest change that addresses the cause — this is the only step that can touch the repo; investigation is read-only.
3. Commits, pushes its branch, and opens a real pull request describing what was wrong and what it changed.

If it isn't confident, it says so and stops rather than guessing — you'll see why in the card instead of a PR link. Either way, nothing reaches your repo's default branch without you reviewing and merging the PR yourself.

Sandbox

Every fix runs inside a fresh, isolated clone — never the live project folder — on its own branch, deleted once the PR is pushed (or the attempt is abandoned). This is enforced server-side in the file tools the agent gets, not something the prompt has to be trusted to honor: it cannot push to the default branch, and it has no access to anything outside its own workspace.

A fix needs the project to have a connected GitHub repo — there's nowhere to open a PR against otherwise; Fix it and open a PR won't appear without one.

Env var overrides

The bottom bar's env-override control (the sliders icon next to the browser picker) applies KEY=VALUE lines on top of the project's own .env, for the next run only. A handful of interpreter-level variable names (NODE_OPTIONS, PATH, LD_PRELOAD and similar) are always rejected, since they'd reach Plum's own process rather than just the test under it.