test(integration): add oracle parity test for JacobianLens (#1539 Tier-1)

[mukund1985]

Closes #1539 Tier-1 item 1.

What

Adds tests/integration/test_jacobian_lens_oracle_parity.py — a @pytest.mark.slow integration test that compares TransformerBridge JacobianLens.readout() against the reference anthropics/jacobian-lens oracle across 75 layer×prompt cells.

Why

#1539 Tier-1 items require a test asserting numerical parity between the TL implementation and the oracle. The oracle is pinned to commit 581d398 (Apache-2.0, frozen) so the threshold is reproducible independent of upstream drift.

How it works

The oracle (jlens.lens.JacobianLens) and TL's JacobianLens both load the same published checkpoint from neuronpedia/jacobian-lens. The oracle is installed at test time via pip from the pinned commit:

git+https://github.com/anthropics/jacobian-lens.git@581d398613e5602a5af361e1c34d3a92ea82ba8e

Both lenses run on google/gemma-2-2b-it. The oracle wraps bridge.original_model and bridge.tokenizer via jlens.hf.from_hf() — no second model copy is loaded.

Pass criteria (per #1539 spec, matching #1505 spike numbers)

75 cells = 5 prompts × 15 sampled layers (0–24 plus final)

• Worst-case top-8 token overlap >= 7/8 in every cell
• Spearman rank-correlation on top-64 logit union >= 0.95 in every cell

Final layer (identity transport) is verified against oracle's model_logits rather than lens output, since the oracle raises if passed a layer not in source_layers.

Test structure

tests/integration/test_jacobian_lens_oracle_parity.py
  oracle_package        fixture — pip-installs pinned oracle, returns (OracleJL, from_hf)
  gemma_bridge          fixture — loads google/gemma-2-2b-it as TransformerBridge
  tl_lens               fixture — JacobianLens.from_pretrained(neuronpedia/jacobian-lens)
  oracle_lens_and_model fixture — oracle JacobianLens + HFLensModel (shared HF model)
  test_jacobian_lens_oracle_parity  @pytest.mark.slow — runs all 75 cells, reports all failures

Spearman is computed inline with numpy (no scipy dependency).

[mukund1985]

@jlarson4 when you get a chance — this is the external-oracle parity test for #1539 Tier-1 item 1. Pinned the oracle to 581d398 so the thresholds are reproducible. Let me know if you want the tolerance numbers adjusted or the prompt set changed.

[jlarson4]

Awesome, thanks @mukund1985! I will make a look at this first thing Monday.