In-Context Collapse in Vision-Language Models

Code to reproduce every result from scratch for the paper Integration-Circuit Adaptation for Mitigating In-Context Collapse in Vision-Language Models.

As more in-context demonstrations are supplied, some vision-language models (VLMs) get worse — their accuracy falls, sometimes below chance, while they still emit well-formed labels. This repository contains the full pipeline that characterizes this in-context collapse, causally localizes it to the vision-language integration pathway, and repairs it with a one-time integration vaccine (CIRCA).

Everything runs locally on a single GPU (or CPU). There is no dependency on any cloud provider, hosted training, or proprietary infrastructure: models download from the Hugging Face Hub and all experiments run in-process.

---

Install

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

torch should match your CUDA version (see https://pytorch.org). Everything also runs on Apple Silicon (MPS) and on CPU without code changes — the device is selected automatically.

---

Two ways to reproduce

The result of every experiment is a JSON file under data/figdata/ (or per-cell files under data/panel_cells/). The figures are then rendered from those JSONs.

• Regenerate the figures from the JSONs already in this repo (no GPU, ~30 s):

  python -m incontext_collapse.figures        # writes all 18 figures/*.pdf

• Regenerate the results from scratch (downloads models, runs on GPU), then the figures. Each driver below writes its JSON into data/figdata/; re-run the figure command afterwards. The shipped JSONs are provided so you can both check your reproduction and rebuild figures without a GPU — they are not required for the drivers to run.

Quick end-to-end sanity check on one small model (downloads ~1 GB, runs anywhere):

python -m incontext_collapse.panel --quick

---

Reproduce each result from scratch

Paper item	Command	Output JSON
Fig. 1 / Table 1 collapse panel + 3 regimes	python -m incontext_collapse.panel --seeds 3	data/panel_cells/*
Fig. 2 / Tables causal lesion-and-rescue	python -m incontext_collapse.circuit --model Qwen/Qwen2-VL-2B-Instruct --task shapes4 --seeds 0,1,2	data/figdata/circuit.json
Fig. 8 / Tables integration vaccine + locus ablation (×3 models)	python -m incontext_collapse.vaccine --model Qwen/Qwen2-VL-2B-Instruct --seeds 0,1,2 --out data/figdata/vaccine.json	data/figdata/vaccine*.json
Table repair-strategy comparison	python -m incontext_collapse.compare --model Qwen/Qwen2-VL-2B-Instruct --seeds 0,1,2	data/figdata/compare.json
CIRCA inject / gate sweep	python -m incontext_collapse.inject --model Qwen/Qwen2-VL-2B-Instruct --modes manyshot,zeroshot,inject	(stdout / --out)
Fig. 12 / Table 3 collapse on VQA benchmarks	python -m incontext_collapse.vqa --model Qwen/Qwen2-VL-2B-Instruct --datasets vqav2,textvqa,scienceqa --seeds 0,1,2 --out data/figdata/vqa_ci_qwen2b.json	data/figdata/vqa_ci_*.json
Fig. 13 VQA failure gallery (per prone model)	python -m incontext_collapse.vqa --model Qwen/Qwen2-VL-2B-Instruct --ks 0,8 --n 120 --gallery --out data/figdata/vqa_gallery_qwen2b_v2.json	data/figdata/vqa_gallery_*.json
Fig. 4 integration ≠ consolidation	python -m incontext_collapse.continual --T 5 --C 4 --seeds 3	data/figdata/continual.json
Fig. 3 frontier-model probe (standard + remap)	python -m incontext_collapse.frontier --model <provider-model-id> --cond standard (needs pip install anthropic and ANTHROPIC_API_KEY)	data/figdata/frontier_*.json

Every driver supports --help. Common flags: --model (any HF VLM id), --seeds (comma-separated; ≥2 gives 95% CIs), --ks (shot grid), --out (result path). Long runs are resumable — re-running skips cells/regions already written.

The default panel spans eleven open VLMs (0.5B–11B) across MLP-projector (Qwen2-VL, Qwen2.5-VL, LLaVA families), pixel-shuffle (SmolVLM), and cross-attention (Llama-3.2-Vision) connectors. Edit panel.DEFAULT_MODELS to add or drop models.

The three CIRCA architectures (Fig. 8, Tables 5–6)

The full vaccine pipeline is run on three models chosen to span two connector families and two backbone lineages. Reproduce each by pointing --out at the filename the figure renderer reads:

python -m incontext_collapse.vaccine --model Qwen/Qwen2-VL-2B-Instruct \
    --seeds 0,1,2 --out data/figdata/vaccine.json            # MLP projector
python -m incontext_collapse.vaccine --model Qwen/Qwen2.5-VL-3B-Instruct \
    --seeds 0,1,2 --out data/figdata/vaccine_qwen25_3b.json  # later-gen MLP
python -m incontext_collapse.vaccine --model OpenGVLab/InternVL3-2B-hf \
    --seeds 0,1,2 --out data/figdata/vaccine_internvl3.json  # pixel-shuffle

The same vaccine driver reproduces the locus ablation (Table 6) via --regions late (equal-capacity readout control) or single-component drops (e.g. --regions early,mid). The lesion-and-rescue (circuit) is likewise architecture-agnostic — pass any of the three model ids.

The frontier probe takes --cond standard (real labels, robustness) or --cond remap (arbitrary tokens, genuine in-context learning); add --firm for verbose instruction-tuned models that otherwise answer in full sentences. Swap frontier.call_model to target any chat provider; the rest of the harness is unchanged.

---

Layout

incontext_collapse/
  core.py        tasks, verbalizers, prompts, inference, region targeting (shared)
  models.py      unified model loading (CUDA / MPS / CPU)
  panel.py       collapse panel + regimes              (Fig. 1)
  circuit.py     causal lesion-and-rescue              (Fig. 2)
  vaccine.py     transferable integration vaccine      (Fig. 8)
  inject.py      CIRCA inject / gate task-vector path
  compare.py     repair-strategy comparison            (Table)
  vqa.py         collapse on VQA benchmarks            (Table)
  continual.py   integration vs. consolidation         (Fig. 4)
  frontier.py    frontier-model probe (provider-neutral, API-only)
  figures.py     render all figures from result JSONs
data/
  figdata/       aggregated result JSONs read by figures.py
  panel_cells/   per-(model,task,condition,K,seed) panel results
figures/         generated PDFs

Method, in brief

A VLM is unembed ∘ language-model ∘ connector ∘ vision-encoder. We split the language-model blocks into thirds and define an integration locus (connector + early/mid) and a readout locus (late). The collapse is localized by a parameter-matched lesion-and-rescue: a small rank-r LoRA adapter on the integration locus restores in-context learning, while an equal-capacity adapter on the readout does not. CIRCA turns this locus into a one-time vaccine — a single adapter trained once on a synthetic task — whose collapse-resistance transfers to unseen tasks. See core.region_targets and circuit.train_region for the exact construction.

Citation

@article{rostami2026incontextcollapse,
  title   = {Integration-Circuit Adaptation for Mitigating In-Context Collapse
             in Vision-Language Models},
  author  = {Rostami, Mohammad},
  journal = {},
  year    = {2026}
}

Released under the MIT License (see LICENSE).