LSP Phase 9: README + CONFIG docs + manual test plan

CONFIG.md

@@ -122,6 +122,62 @@ adds a default Exa Web Search MCP server at `https://mcp.exa.ai/mcp` with the
 `x-api-key` header set to `EXA_API_KEY` when that environment variable is set.
 Set `"mcp_servers": {}` to disable all MCP servers.
 
+## LSP configuration
+
+When compiled with the `lsp` feature (default-on), dirge spawns language
+servers on demand to surface compile errors in tool output. The `lsp` config
+key accepts three forms:
+
+```json
+// Default-on, built-in commands for rust/typescript/pyright/clojure-lsp.
+{ "lsp": true }
+
+// Off entirely. Same as the --no-lsp CLI flag.
+{ "lsp": false }
+
+// Default-on with per-server overrides.
+{
+  "lsp": {
+    "rust": {
+      "command": ["rust-analyzer"],
+      "env": { "RA_LOG": "rust_analyzer=debug" },
+      "initialization": { "cargo": { "buildScripts": { "enable": true } } }
+    },
+    "typescript": { "disabled": true }
+  }
+}
+```
+
+Per-server fields (all optional):
+
+| Field            | Type             | Description |
+| ---------------- | ---------------- | ----------- |
+| `command`        | string[]         | argv to launch the server. Replaces the built-in default. |
+| `extensions`     | string[]         | *Reserved.* Currently ignored — see "Known limitations" below. |
+| `env`            | object           | extra env vars for the child process. |
+| `initialization` | object           | sent as `initializationOptions` in the LSP `initialize` request. |
+| `disabled`       | boolean          | `true` removes the server entirely. |
+
+CLI flag: `--no-lsp` (overrides the config; same effect as `lsp: false`).
+
+### Built-in server commands
+
+| Server id     | Default command                              |
+| ------------- | -------------------------------------------- |
+| `rust`        | `rust-analyzer`                              |
+| `typescript`  | `typescript-language-server --stdio`         |
+| `pyright`     | `pyright-langserver --stdio`                 |
+| `clojure-lsp` | `clojure-lsp`                                |
+
+Servers are spawned lazily on first file touch and cached per `(workspace_root, server_id)` pair. Concurrent agent tool calls for the same file deduplicate so dirge never races two `rust-analyzer` processes against one workspace.
+
+### Known limitations
+
+- The `extensions` override is currently ignored. The claimed-extensions list lives in the static `builtin_servers()` registry at `src/lsp/server.rs`. Adding new extensions today requires editing that file. Follow-up.
+- v1 has four built-in servers. Additional servers can be added by extending `builtin_servers()` + `ProcessSpawner::default_commands()` in source.
+
+For an end-to-end smoke test against a real `rust-analyzer` process, see [`docs/LSP_MANUAL_TEST.md`](docs/LSP_MANUAL_TEST.md).
+
 ## ACP (Agent Communication Protocol) configuration
 
 When compiled with the `acp` feature, dirge can act as an ACP agent server.

README.md

@@ -255,6 +255,30 @@ Create `~/.config/dirge/plugins/my-plugin.janet`:
               "\n- Authentication bypass"))))
 ```
 
+## LSP integration
+
+When built with the `lsp` feature (on by default), dirge attaches Language Server Protocol clients to your project and surfaces compile-time diagnostics directly in the agent's tool output. After every `write` or `edit`, the LSP server gets a `didChange`, waits for a fresh diagnostic publish, and any ERRORs land in the tool result as a `<diagnostics file="...">` block — so the agent corrects compile errors on the same turn instead of writing broken code and discovering it later via `cargo check`.
+
+| Tool | Effect |
+|------|--------|
+| `read`  | Fire-and-forget `didOpen` so the server has the file in memory by the time the agent edits it. No diagnostic block in `read` output. |
+| `write` | After write: `didChange` + wait for diagnostics + append errors-block. |
+| `edit`  | Same as `write`. |
+| `lsp`   | Agent-facing tool that exposes `definition`, `references`, `hover`, `documentSymbol`, `workspaceSymbol`, `implementation`, `prepareCallHierarchy`, `incomingCalls`, `outgoingCalls`. 1-based coordinates. |
+
+Built-in server set:
+
+| Server id | Binary | Extensions |
+|-----------|--------|------------|
+| `rust` | `rust-analyzer` | `.rs` |
+| `typescript` | `typescript-language-server --stdio` | `.ts`, `.tsx`, `.mts`, `.cts`, `.js`, `.jsx`, `.mjs`, `.cjs` |
+| `pyright` | `pyright-langserver --stdio` | `.py`, `.pyi` |
+| `clojure-lsp` | `clojure-lsp` | `.clj`, `.cljs`, `.cljc`, `.edn`, `.bb` |
+
+Workspace root resolution is per-server: rust-analyzer walks past nested member crates to the workspace `Cargo.toml` declaring `[workspace]`; typescript stops at the nearest `package.json`/`tsconfig.json` and yields to deno when a `deno.json` is closer; pyright looks for `pyproject.toml`/`setup.py`/etc.; clojure-lsp looks for `deps.edn`/`project.clj`/`shadow-cljs.edn`/`bb.edn`/`.clj-kondo`.
+
+Disable: `--no-lsp` flag or `{ "lsp": false }` in the config. Per-server overrides (custom command, env, init options) live in the config — see [CONFIG.md](CONFIG.md).
+
 ## Semantic code tools
 
 When built with `--features "semantic,semantic-ts,semantic-python"`, dirge gains AST-powered code analysis:

docs/LSP_MANUAL_TEST.md

@@ -0,0 +1,132 @@
+# LSP integration — manual end-to-end test
+
+The LSP integration is covered by ~120 unit tests against mock spawners and
+duplex pipes. This document captures the manual smoke test against a real
+`rust-analyzer` process — the one thing CI can't reproduce.
+
+## Prerequisites
+
+- `rust-analyzer` on `$PATH`. Verify with:
+  ```bash
+  which rust-analyzer && rust-analyzer --version
+  ```
+  If absent, `rustup component add rust-analyzer` from the default toolchain.
+- A Rust project on disk. The dirge repo itself works.
+
+## Scenario 1 — diagnostic surfacing on edit
+
+**Goal**: after a deliberately broken edit, the `edit` tool's output
+contains a `<diagnostics file="...">` block with the compile error.
+
+1. Start dirge in the dirge repo: `cargo run --release`.
+2. Wait for the initial prompt.
+3. Type:
+   ```
+   Edit src/agent/builder.rs and change `pub async fn build_agent_inner` to
+   `pub async fnn build_agent_inner` (typo: fnn instead of fn). After the
+   edit, show me the exact tool output you got back.
+   ```
+4. **Expected behavior**:
+   - The agent calls the `edit` tool, the file is mutated.
+   - Within ~10 seconds (the `DIAGNOSTIC_WAIT` constant), the agent receives
+     the tool result containing a section like:
+     ```
+     LSP errors detected in this file, please fix:
+     <diagnostics file="/Users/.../src/agent/builder.rs">
+     ERROR [N:M] expected one of `(`, `[`, `;`, or `<`, found `build_agent_inner`
+     ...
+     </diagnostics>
+     ```
+   - The agent then proposes a corrective edit (`fn` not `fnn`).
+5. Ctrl+C to abort, then revert: `git checkout src/agent/builder.rs`.
+
+**Failure modes to watch for**:
+- No diagnostic block → check `rust-analyzer` is on PATH and the project has
+  built at least once.
+- Diagnostic block but with WARN entries → bug: only ERRORs should surface.
+- Block appears but takes >15 seconds → bug: bounded wait isn't firing.
+
+## Scenario 2 — `lsp` tool: hover at a known position
+
+**Goal**: `lsp` tool dispatches `hover` and returns server-side info.
+
+1. From inside dirge:
+   ```
+   Use the lsp tool to hover at src/main.rs line 1 character 1.
+   ```
+2. **Expected behavior**:
+   - The agent calls `lsp` with
+     `{"operation": "hover", "file_path": "src/main.rs", "line": 1, "character": 1}`.
+   - Result is pretty-printed JSON from rust-analyzer: a `contents` field
+     with the hovered-token info. Even at "mod agent;" position 1, the
+     server may return null hover info; the tool reports
+     `(no results from hover)`.
+   - **Validation**: cursor at a real identifier (say, "build_agent" inside
+     `src/main.rs`) should return non-empty hover content.
+
+## Scenario 3 — workspace symbol search
+
+**Goal**: `lsp.workspaceSymbol` returns matches across the workspace.
+
+1. From inside dirge:
+   ```
+   Use the lsp tool with workspaceSymbol operation to find symbols
+   matching "build_agent_inner". Pass src/main.rs as file_path (just to
+   pick the workspace).
+   ```
+2. **Expected behavior**:
+   - Pretty-printed JSON array with at least one entry pointing to
+     `src/agent/builder.rs`.
+
+## Scenario 4 — concurrent file edits don't spawn duplicate servers
+
+**Goal**: the inflight-spawn dedupe works end-to-end.
+
+1. Open two dirge sessions in different terminals, same repo.
+2. In both, ask the agent to read `src/main.rs`. The first read triggers a
+   spawn; the second should reuse the cached client (verifiable via
+   `ps aux | grep rust-analyzer` — only one process per workspace within
+   a single dirge session; across sessions there's one process per).
+3. **Expected behavior**: across the lifetime of one dirge session,
+   `ps aux | grep rust-analyzer | wc -l` returns 1 (the workspace root
+   stays the same, so one server services all .rs touches).
+
+## Scenario 5 — `--no-lsp` actually disables
+
+**Goal**: feature gate / CLI flag work.
+
+1. Run `cargo run --release -- --no-lsp` and ask the agent to edit a Rust
+   file. The tool output must NOT contain any `<diagnostics>` block.
+2. Run `cargo run --release --no-default-features --features 'loop git-worktree mcp'`
+   and verify the binary builds and runs. The `lsp` tool and diagnostic
+   block both should be absent.
+
+## Scenario 6 — broken spawn doesn't retry
+
+**Goal**: failed spawn marks (root, server_id) as broken so subsequent
+file touches don't re-spawn.
+
+1. Disable rust-analyzer temporarily:
+   ```bash
+   mv "$(which rust-analyzer)" "$(which rust-analyzer).bak"
+   ```
+2. Start dirge, ask to read a `.rs` file. First touch triggers a spawn;
+   it fails (binary missing).
+3. Ask to read another `.rs` file. Watch for log lines (run with
+   `RUST_LOG=warn`): the second read should NOT log "spawn failed" — the
+   broken-set blocks the retry.
+4. Restore: `mv "$(which rust-analyzer).bak" "$(which rust-analyzer)"`.
+
+## What this test plan deliberately doesn't cover
+
+These are exercised by unit tests against the mock spawner and don't need
+manual verification:
+- JSON-RPC framing edge cases (multi-message buffers, partial reads).
+- Request correlation by id under concurrent in-flight requests.
+- Diagnostic dedupe + MAX_PER_FILE caps.
+- Push-vs-pull diagnostic merge.
+- URI ↔ path round-tripping with special characters.
+- Config schema parsing.
+
+See the corresponding tests under `src/lsp/**/tests::` for the contracts
+those features guarantee.