🧩 AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Models

🔗 Codebase   |   📑 Paper (coming soon)

AgentPatch is a training-free framework for merging specialized agentic multimodal large language models (MLLMs) into a single generalist checkpoint. It repairs capabilities that are weakened during merging while preserving complementary search, GUI-interaction, and agentic visual-processing skills.

This repository provides the official implementation of our paper:

AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Models

---

📰 News

• [2026/08/06] We release AgentPatch, including its core repair operators, DGC evidence interface, configuration example, and CPU tests.
• The paper, complete checkpoint-construction pipeline, benchmark adapters, and model checkpoints will be released progressively.

🚀 Features

• 🧩 Agentic MLLM Merging. AgentPatch consolidates experts for multimodal search, GUI interaction, and agentic visual processing into one static checkpoint.
• ⚖️ Stable Backbone Selection. Candidate training-free merges are evaluated on development subsets to select a reliable recipient and identify its weak task.
• 🔧 Weak-Task Unique Residual Recovery. Parameter updates unique to the weak-task expert are selectively restored to provide coarse capability recovery.
• 🤖 Agent-Guided Behavior-Critical Patch. The Diagnoser–Guardian–Compiler (DGC) workflow compiles repair and protection behavior spans from recipient–expert trajectories.
• 🛡️ Guardian-Constrained Repair. Behavior-critical FFN neurons are localized using activation and output contribution, while protected recipient neurons are excluded from interpolation.
• ⚡ Training-Free and Deployment-Friendly. AgentPatch requires no additional gradient training, expert routing, ensemble inference, or analysis agent at test time.

---

📦 Installation

AgentPatch requires Python 3.10+ and PyTorch 2.3+.

python -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
python -m pip install -e '.[dev]'

Run the CPU tests to verify the installation:

python -m pytest tests

---

🏗️ Running AgentPatch

Prepare the recipient checkpoint, aligned expert checkpoints, and the offline DGC evidence bundle described in docs/input_contract.md. Validate the inputs with:

bash scripts/validate_dgc_inputs.sh /path/to/dgc_inputs

The core repair operators can then be imported into the checkpoint-construction pipeline:

from agentpatch import (
    build_unique_mask,
    exclude_protected_neurons,
    interpolate_ffn_neurons,
    recover_unique_residual,
    score_behavior_critical_neurons,
)

See docs/reproduction.md for the three-stage workflow and src/agentpatch/core.py for the tensor conventions of each operator.

---

📈 Evaluation

Our paper evaluates AgentPatch on six agentic and multimodal benchmarks spanning three capability families:

• Agentic multimodal search: MMSearch and FactualVQA;
• GUI interaction: AndroidWorld and OSWorld;
• Agentic visual processing: V*Bench and HR-Bench 8K.

AgentPatch produces one static merged checkpoint and uses neither expert routing nor ensemble inference during evaluation. The complete benchmark adapters and path-free evaluation commands will be added with the full reproduction release. See docs/reproduction.md for the current stage-by-stage reproduction protocol.

---

📊 Results

Across the six benchmarks, AgentPatch improves the overall average of its TSVM recipient from 54.5 to 56.6. It strengthens both mobile and desktop interaction while preserving complementary multimodal search and visual-processing capabilities. AgentPatch also improves alternative merged recipients, showing that its repair mechanism is not tied to a single backbone construction rule.

---

🗂️ Repository Structure

AgentPatch/
├── assets/
│   └── framework.png              # AgentPatch framework
├── configs/
│   └── agentpatch.example.yaml    # Paper-aligned configuration
├── docs/
│   ├── input_contract.md          # DGC evidence format
│   └── reproduction.md            # Reproduction protocol
├── scripts/
│   └── validate_dgc_inputs.sh     # DGC input validation
├── src/agentpatch/
│   ├── core.py                    # Core repair operators
│   └── workflow/                  # DGC contracts and selectors
├── tests/                         # CPU unit tests
└── pyproject.toml                 # Package metadata and dependencies

---

📄 Citation

If you find AgentPatch useful, please cite our work:

@article{shao2026agentpatch,
  title={AgentPatch: Coarse-to-Fine Weak-Task Repair for Merging Agentic Multimodal Large Language Models},
  author={Shao, Zibo and Xiong, Baochen and Xu, Chengdong and Xiao, Linhui and Li, Kaichen and Gong, Haoran and Li, Yan and Song, Yaguang and Yang, Xiaoshan},
  journal={arXiv preprint},
  year={2026}
}

---

📜 License

AgentPatch is released under the Apache License 2.0. This repository does not redistribute model checkpoints, benchmark data, trajectories, or evaluation assets; obtain those resources from their official sources and follow their respective licenses and terms.

---

🔍 Acknowledgements

We thank the developers of the expert models, model-merging methods, and agentic evaluation environments used in this project, including Qwen2.5-VL, MMSearch-R1, GUI-Owl, DeepEyes, AndroidWorld, OSWorld, V*Bench, and HR-Bench 8K. Please refer to the corresponding upstream projects for their licenses and usage requirements.