test(acp): establish a Devin CLI compatibility contract

[errfld]

Tracked by #6.

Why this issue exists

Buzz already launches arbitrary ACP-over-stdio agents, and upstream PR block#3225 added a tier-2 Devin preset. Before adding first-class metadata or UI behavior, pin the current native devin acp contract so later issues do not guess at vendor behavior or introduce unnecessary adapter code.

The integration seam is native ACP. Do not adapt the interactive TUI or devin --print; neither provides the structured, bidirectional lifecycle Buzz already gets from ACP.

Current verified baseline

A local read-only probe against devin 3000.3.22 observed:

• command devin acp using newline-delimited JSON-RPC
• initialize response protocol version 1
• agentCapabilities.loadSession = true
• browser and terminal auth methods (devin-browser, devin-terminal-login)
• no steering capability; Buzz must retain its cancel-and-merge fallback
• modes accept-edits, smart, ask, plan, and bypass
• model/mode options using fields id, name, and currentValue

Treat these as starting hypotheses. Runtime-advertised capabilities remain authoritative.

Likely touchpoints

• crates/buzz-acp/src/acp.rs: initialize, session_new_full, model extractors/switching, permission requests
• crates/buzz-acp/src/lib.rs: run_auth_methods, run_authenticate, run_models
• crates/buzz-acp/src/pool.rs: session/model/mode setup, cancellation, non-steering follow-ups
• desktop/tests/e2e/fixtures/fake-acp-agent.mjs
• Add the durable redacted result as docs/devin-cli-compatibility.md.

Prefer extending existing fake-agent/script fixtures over introducing a new framework.

Work

1. Exercise a currently supported Devin CLI through the same JSON-RPC messages Buzz sends.
2. Record initialize/auth, session creation/config, streaming, permissions, cancellation, stop reasons, optional MCP use, EOF, and capability-gated session load.
3. Confirm follow-up behavior when steering is absent.
4. Reproduce or close out the early-visible-result/session-completion behavior described in feat(desktop): add native Devin ACP runtime block/buzz#3072.
5. Convert every stable compatibility assumption needed by Buzz into a hermetic fake-agent regression.
6. Keep real authenticated turns manual or environment-gated; CI must never require Devin.

Acceptance criteria

• docs/devin-cli-compatibility.md records tested versions, capabilities, wire shapes, and results without secrets.
• Fixtures cover Devin's protocol-v1 response and id/name/currentValue config shape.
• Cancellation, missing steering, permission requests, and subprocess exit have recorded outcomes.
• Any Devin-specific workaround has a minimal reproduction and regression test; otherwise the bridge remains generic.
• No credentials, prompts, account identifiers, model entitlements, or unredacted vendor errors are committed or logged.
• A minimum supported version is documented only if a real incompatibility requires one.

Suggested verification

. ./bin/activate-hermit
cargo test -p buzz-acp

# Safe probes; do not commit auth output.
devin --version
devin auth status >/dev/null
BUZZ_ACP_AGENT_COMMAND=devin BUZZ_ACP_AGENT_ARGS=acp cargo run -p buzz-acp -- auth-methods --json
BUZZ_ACP_AGENT_COMMAND=devin BUZZ_ACP_AGENT_ARGS=acp cargo run -p buzz-acp -- models --json

Authenticated prompt/reply smoke testing is optional and must be documented as potentially billable.

Out of scope

• Runtime catalog/UI changes (feat(desktop): promote Devin to a first-class managed ACP runtime #2 and feat(desktop): add Devin onboarding, models, and config integration #3)
• Cloud Devin or the remote Devin MCP service
• A static model list
• Generic session-resume implementation beyond documenting advertised support

References

• https://docs.devin.ai/cli/reference/commands
• https://docs.devin.ai/cli/acp/jetbrains
• https://agentclientprotocol.com/protocol/v1/transports
• https://cdn.agentclientprotocol.com/registry/v1/latest/registry.json