feat!: replace LSP oracle with SCIP indexers (TS/Py/Go/Rust/Java)

[theagenticguy]

Summary

• Rip @opencodehub/lsp-oracle and the four per-language LSP upgrade phases; replace with a single scip-index phase backed by the new @opencodehub/scip-ingest package. Full coverage of TypeScript, Python, Go, Rust, and Java via the native SCIP indexers (scip-typescript, scip-python, scip-go, rust-analyzer --scip, scip-java).
• Gym replay harness migrated to the same SCIP rails (ScipClient runs the indexer once, pre-builds in-memory occurrence + definition lookups, answers queryCallers/References/Implementations without re-decoding).
• New output-side codehub pack CLI + pack_codebase MCP tool wrapping repomix --compress. ADR 0005 documents why repomix cannot replace the tree-sitter input-side chunker (per-file text blobs, tokenizer mismatch, missing tsx/kotlin coverage).

BREAKING: oracle-edge provenance prefixes change from {pyright,typescript-language-server,gopls,rust-analyzer}@<ver> to scip:<indexer>@<ver>. The +lsp-unconfirmed demoted-reason suffix becomes +scip-unconfirmed. LSP_PROVENANCE_PREFIXES is kept as a deprecated alias for SCIP_PROVENANCE_PREFIXES for one release.

Numbers

• Net −12,037 LOC (−15,572 deletions, +3,535 additions) across 81 files.
• 1,414/1,414 tests pass across 14 packages.
• pnpm run check clean (lint + typecheck + tests + banned-strings).
• End-to-end smoke on the bundled calcpkg.scip fixture: add() tops blast ranking with 9 backward callers — matches the reference POC ranking.

Architecture

source tree
  ├─ tree-sitter parse  (unchanged: scan / parse / structure /
  │                       accesses / cross-file / mro / complexity)
  └─ scip-index phase   (NEW)
       ├─ detectLanguages(repoRoot)
       ├─ runIndexer(lang, …)  → .opencodehub/scip/<lang>.scip
       ├─ parseScipIndex(buf)  → ScipIndex
       ├─ deriveIndex(index)   → {symbols, edges}
       └─ CodeRelation(confidence=1.0, reason=\"scip:<indexer>@<ver>\")
            │
            ▼
       confidence-demote  → summarize / mcp-confidence / analyze-auto-cap

Follow-ups (non-blocking)

Captured in .erpaval/sessions/session-f8a300bc/lessons.yaml:

• Regenerate gym corpus baselines (packages/gym/corpus/**/*.yaml, baselines/manifest.jsonl) against SCIP indexers.
• Retarget .github/workflows/gym.yml from gopls/pyright/tsserver/rust-analyzer caches to scip-* caches.
• Supersede docs/adr/0003-ci-toolchain-pins.md with a SCIP-indexer pin ADR.
• Extend the scip-ingest parser to decode SymbolInformation.relationships so IMPLEMENTS/EXTENDS edges are first-class (today the minimal reader best-efforts them).

Test plan

• Manual: pnpm install && pnpm run check on a clean clone.
• Manual: pnpm --filter @opencodehub/scip-ingest test against the bundled calcpkg.scip fixture (6 tests, covers parse / derive / materialize).
• Manual: codehub analyze on a TS monorepo (requires scip-typescript on PATH). Expect scip-index phase to emit edges with reason: scip:scip-typescript@<ver>.
• Manual: codehub pack on any repo — produces .codehub/pack/repo.xml.
• Manual: MCP pack_codebase tool via codehub mcp stdio.
• CI: full pnpm -r test matrix green (gym.yml changes deferred per follow-ups — existing LSP-matrix workflow may break on this branch; that is expected and captured in the follow-up list).

Risk

• Gym corpus baselines were captured against the old LSP clients. They still load and still drive the new SCIP replay harness, but per-case expected result sets need regeneration. Failures on the gym.yml workflow today are content-level, not structural. Mitigated by not running gym replay in required CI for this PR.
• scip-java / rust-analyzer execute build scripts during indexing. Gated behind CODEHUB_ALLOW_BUILD_SCRIPTS=1 on untrusted workspaces; skip reason surfaces in the phase output otherwise.

[github-actions]

Gym regression: typescript

The gym gate failed for the typescript matrix cell on commit 13b55a5.

Rollup summary from this run:

(no rollups in manifest)

Full manifest artifact: gym-manifest-typescript-13b55a5bd100328d2ae9c7b7faea7ce2db5089df (uploaded on failure).

If this regression is intentional, update packages/gym/baselines/manifest.jsonl in a follow-up PR using mise run gym:baseline.

[github-actions]

Gym regression: go

The gym gate failed for the go matrix cell on commit 13b55a5.

Rollup summary from this run:

(no rollups in manifest)

Full manifest artifact: gym-manifest-go-13b55a5bd100328d2ae9c7b7faea7ce2db5089df (uploaded on failure).

If this regression is intentional, update packages/gym/baselines/manifest.jsonl in a follow-up PR using mise run gym:baseline.

[github-actions]

Gym regression: rust

The gym gate failed for the rust matrix cell on commit 13b55a5.

Rollup summary from this run:

(no rollups in manifest)

Full manifest artifact: gym-manifest-rust-13b55a5bd100328d2ae9c7b7faea7ce2db5089df (uploaded on failure).

If this regression is intentional, update packages/gym/baselines/manifest.jsonl in a follow-up PR using mise run gym:baseline.

[github-actions]

Gym regression: rust

The SCIP gym gate failed for the rust matrix cell on commit 61c1885.

Rollup summary from this run:

(no rollups in manifest)

Full manifest artifact: gym-manifest-rust-61c18858949dfa44c04766581bd6cd46561784a0 (uploaded on failure).

If this regression is intentional, update packages/gym/baselines/manifest.jsonl via mise run gym:baseline and commit the refresh.