Idea: add LSP (language-server) tooling to the dev-lead / pr-review agent harness

[don-petry]

TL;DR

Adding LSP (Language Server Protocol) tooling to our agent harness is worth piloting — it gives the agents real semantic code intelligence (go-to-definition, find-references, type resolution, diagnostics) instead of grep + file-reading. The clearest wins are dev-lead writing/fixing code and pr-review verifying findings across files. The one catch unique to us: LSP servers must cold-start and index on every ephemeral CI run, so it needs a caching + tier-gating strategy. It would not have helped the config/YAML/secrets incidents we hit recently — LSP raises code-intelligence, not config-intelligence.

---

The problem it solves (in our terms)

Our agents navigate code by grep + reading files. That fails exactly where they operate:

• Cross-file evidence chains — a review or CI-fix that follows a symbol through imports/types across the repo. Grep returns textual matches (validateToken in a string/comment), not semantic ones.
• Third-party / transitive types — resolving a library signature means walking node_modules/site-packages or guessing → a documented source of hallucinated APIs (~20% of AI package suggestions point at libraries that don't exist).
• Context-window slicing — large repos don't fit in context, so the agent gets a "sliced, incomplete, often misleading view." Our batch reviewer and dev-lead fixers hit this constantly.

LSP replaces that with textDocument/definition, references, hover, diagnostics, documentSymbol, rename — exact source locations + fully-resolved types, exposed to the agent as MCP tools.

Quantified benefits

• Token efficiency, 5–34× vs grep/read on the same task. Cited blast-radius analysis on 319K-line Consul: 17.7 MB → 841 KB, 5,534 tool calls → 119. Find-references in a 100-file project ~500 tokens (LSP) vs 2,000+ (grep).
• ~20–30 min of manual file-tracing saved per error (TypeScript).
• Accuracy: a validation phase that cross-checks each finding against real codebase context is what separates "confidently wrong" from "evidence-based" review — and LSP is that validator.

Why this matters to us specifically: we run a Token Cost Observatory and just had a Gemini-credits-depleted incident. A 5–34× cut on navigation-heavy turns is a direct hit to per-run token spend and rate-limit pressure.

Where it helps in our harness

• dev-lead (writing code) — strongest win. Correct signatures on the first pass, safe rename, diagnostics to self-check before pushing → fewer broken commits.
• pr-review (reviewing code) — high value, different shape. Use LSP as a grounding/verification step in the deep/audit tiers: before posting "this is undefined / breaks N callers," confirm via lsp_get_diagnostics / lsp_find_references. Directly attacks the false-positive problem (30–42% of AI-review false positives come from misreading context) — which is exactly what erodes trust in an auto-approving agent.
• Reviewing our own Shell. Our biggest "language" is the harness itself (engine.sh, review-batch.sh, compliance-audit.sh). bash-language-server + existing shellcheck give the agents real navigation over their own substrate.

Costs / caveats specific to us

• Ephemeral runners are the catch. The vendor write-ups assume a persistent, warm-indexed server (IDE/CLI). We run per-PR, time-boxed jobs (60-min cap; triage 300 s / deep 600 s tiers). An LSP server (tsserver, gopls, jdtls) must cold-start + index on every run — minutes on a big repo, eating tier budgets. This is the make-or-break operational issue. Mitigations: actions/cache the LSP index, scope to changed files, gate LSP to deep/audit only, lazy single-file diagnostics.
• Multi-engine is per-engine. LSP reaches the agent as MCP tools. Claude (primary) and Copilot CLI (now Go-live improvements for PR review agent #2 after the fallback-order fix) support MCP/skills; Gemini CLI's MCP support is weakest — degrades on our last-resort engine.
• Multi-language management. Per-language servers to install/version on the runner. Fleet is JS/TS/Python/Shell (manageable); Google Apps Script has no first-class LSP.
• Overlap with existing tooling. We already run SonarCloud, CodeQL, shellcheck. LSP's distinct value is interactive semantic navigation during the agent's reasoning, not batch linting — complementary; scope so we don't pay twice.
• Honest limit: LSP would not have prevented this cycle's incidents (broken @v1 workflow ref, channel-tag pins, the markets PAT override) — those are YAML/config/secrets, outside LSP's wheelhouse.

Recommendation — phased, low-regret

1. Pilot dev-lead on one language/repo (TypeScript in markets, or Shell in .github-private) using an existing LSP-MCP server (e.g. Serena, or the 24-tool CI-verified lsp MCP server) — don't build one.
2. Gate LSP to deep/audit tiers and cache the index to keep cold-start off the critical path.
3. Add a pr-review verification hook that resolves findings via lsp_get_diagnostics/lsp_find_references before posting — measure false-positive rate before/after.
4. Measure with the Token Cost Observatory: tokens/run, tool-call count, review precision — pilot vs control. The 5–34× claim is testable on our own repos.
5. Go/no-go fleet-wide on the data. (MCPBench caution: "using MCPs does not always show noticeable improvement vs plain function-calls" — the measured pilot matters; don't roll out on the vendor narrative alone.)

Open questions for discussion

• Start with dev-lead (code-writing) or pr-review (finding-verification) first?
• Is the per-run indexing cost acceptable, or do we need a persistent/warm LSP service (e.g. a sidecar) before this is viable on CI?
• Which pilot language/repo gives the cleanest signal — TS in markets or Shell in .github-private?

---

Sources

• github.blog — Copilot CLI + language servers
• MindStudio — LSP + Claude Code on large codebases
• Serena MCP server · LSP MCP (24-tool, CI-verified)
• MCPBench evaluation (arXiv) · CR-Bench: AI code-review utility (arXiv)
• Agentic AI code review: confidently-wrong → evidence-based · Diagnosing AI-review false positives

_{🤖 Researched & drafted with Claude Code}

[github-actions[bot]]

Weekly Update — Deep Research: LSP-MCP Server Selection, Caching Design & Tier Gating

What Changed

Major new market signals since initial writeup:

1. GitHub Copilot CLI shipped an LSP Setup skill (June 10, 2026) — the platform vendor now considers LSP-backed code intelligence table stakes for agentic coding. This validates the direction and adds urgency: if we don't add LSP, our agents will be behind the Copilot CLI baseline.

2. agent-lsp (Blackwell Systems) emerged as a strong alternative to Serena and lsp-mcp:

   • Single Go binary, 65 tools + 24 skills, 30 CI-verified languages
   • Persistent daemon with ~10s first-index warm-up, auto-exits after 30min
   • Speculative execution: preview_edit simulates changes in memory before writing — the agent sees what breaks before touching disk
   • GCF (Graph Compact Format): 30–92% token reduction vs JSON responses (92.7% savings by the 5th call with session deduplication)
   • --non-interactive init and --http mode support CI scripting
   • github.com/blackwell-systems/agent-lsp | blog.blackwell-systems.com

3. ManoMano's Project AEGIS benchmark (Feb 2026, 36K-line Java codebase):

   • Vanilla Claude failed a complex refactoring task in 60 min
   • Claude Code + built-in LSP: build failed, 9/1017 tests broken
   • Claude + Serena MCP: 45 min, all 1,017 tests passing, $27.30 (vs $23.54 for vanilla Claude that failed)
   • Key insight: LSP's value is not cheaper per-token — it's avoiding expensive failures
   • ManoMano Tech — Project AEGIS

4. TypeScript 7.0 beta (April 2026) ships tsgo — a Go-native LSP server that is 10x faster than tsserver. Editor startup drops from 9.6s → 1.2s. This largely eliminates the cold-start concern for TypeScript projects on ephemeral runners.

5. MCP token overhead quantified (MindStudio 2026): each MCP tool definition costs ~200 tokens/turn. With 65 tools, that's ~13,000 tokens overhead per turn. Mitigation: expose only 8-12 navigation tools → ~2,000 tokens overhead. Source

6. Gemini CLI status: MCP support exists but is the weakest of the three engines. Unpaid tier transitions to Antigravity CLI on June 18. Our fallback engine should degrade gracefully — skip MCP tools entirely when running on Gemini/Copilot fallback.

Research Answers to Open Questions

Q1: Start with dev-lead (code-writing) or pr-review (finding-verification) first?

Recommendation: pr-review deep tier first. Rationale:

• pr-review runs on every PR — higher signal volume for pilot data
• The finding-verification use case (confirm via find_references/get_diagnostics before posting) is lower risk than code-writing: a bad verification downgrades a finding, a bad code write breaks a build
• The Token Cost Observatory already captures per-PR metrics — pilot vs control comparison is immediate
• dev-lead follows once pr-review pilot validates the indexing cost on real workloads

Q2: Per-run indexing cost — acceptable or need persistent/warm LSP service?

Acceptable with mitigations, no sidecar needed. Key data points:

• agent-lsp: ~10s first-index for typical projects, instant on warm daemon
• tsgo: 1.2s cold start (TS projects are essentially free)
• bash-language-server: lightweight, Tree Sitter based, instant for shell-only repos
• gopls v0.12+: incremental indexing, 75% memory reduction
• 10s indexing on a 600s deep-tier budget = 1.7% overhead — easily acceptable
• Caching strategy: actions/cache keyed on ${{ runner.os }}-lsp-${{ hashFiles('**/package-lock.json', '**/go.sum') }}-${{ env.LSP_SERVER_VERSION }}, matching the existing Claude CLI binary cache pattern (pr-review.yml:197-201)

Q3: Pilot language/repo — TS in markets or Shell in .github-private?

Start with Shell in .github-private (this repo). Rationale:

• This repo is the agent's own substrate — it reviews its own scripts (engine.sh, review-batch.sh, etc.)
• bash-language-server is lightweight and fast — minimal indexing overhead
• Self-hosting the pilot means faster iteration (no cross-repo coordination)
• The review pipeline for this repo already exercises all tiers (triage→deep→duck→audit)
• Follow up with TypeScript (markets) once Shell pilot validates the architecture

Q4: Which LSP-MCP server — Serena vs lsp-mcp vs agent-lsp?

Recommendation: agent-lsp. Comparative analysis:

Criterion	Serena	lsp-mcp	agent-lsp
Language	Python (uv)	Go	Go
CI verification	Not documented	7 languages	30 languages, every push
Cold start	Not documented	Not documented	~10s, daemon persists
Token optimization	N/A	JSON only	GCF: 30-92% reduction
Speculative execution	No	No	Yes (preview_edit, simulate_chain)
Tool count	~15	~24	65 tools + 24 skills
Binary distribution	Python wheel (uv)	Go binary	Go binary
CI scripting	Not recommended	Possible	--non-interactive, --http
Benchmark data	AEGIS ($27.30, 45min, success)	None	None

Serena has the production benchmark (AEGIS), but agent-lsp's CI-first design (Go binary, non-interactive mode, GCF token compression, 30-language CI matrix) better fits our ephemeral-runner, multi-engine architecture. Pilot agent-lsp; use Serena results as the success bar to beat.

Updated Assessment

Dimension	Previous	Current	Delta
Feasibility	high	high	→
Impact	high	high	→ (confirmed by AEGIS benchmark + Copilot CLI LSP skill)
Urgency	med	high	↑ (Copilot CLI shipped LSP June 10; platform is moving)

Recommendation

Advance — start the pilot. All five open questions from the original writeup now have data-backed answers. The integration path is clear: add --mcp-config to run_agentic() in engine.sh for deep/audit tiers only, install agent-lsp Go binary in runner setup, cache the index via actions/cache. Three companion proposals filed as separate Ideas discussions:

1. 💡 LSP-Grounded Finding Verification — the specific pr-review false-positive reduction mechanism
2. 💡 Per-Tier MCP Tool Gating — the engine.sh architecture for selective tool exposure
3. 💡 Token Cost Observatory LSP Metrics — the measurement framework for go/no-go

Sources

• GitHub Blog — Copilot CLI + Language Servers (June 10, 2026)
• agent-lsp (Blackwell Systems) | Blog post
• ManoMano — Project AEGIS Benchmark
• TypeScript 7.0 Beta — tsgo 10x faster
• MindStudio — MCP Token Overhead
• arXiv:2604.10800 — Verify Before You Fix: Execution Grounding
• SonarQube Agentic Analysis MCP Server
• Serena MCP Server
• Gemini CLI MCP Server docs

_{🤖 Researched & drafted by Mary (BMAD Strategic Business Analyst) with Claude Code}