Our investigation reveals that MLLM hallucination involves a predictable two-stage degradation of visual attention during generation:

1. Strong Visual Anchoring Phase — early tokens are firmly grounded in query-relevant visual evidence, with concentrated and adaptive cross-attention patterns.

2. Prior-Dominated Generation Phase — as output length increases, cross-attention progressively decays: it becomes unfocused (spread across many tokens), spatially biased (collapsed to irrelevant regions), and less responsive to newly generated text. The model increasingly relies on language priors, making hallucinations more probable in later tokens.

This degradation manifests in three measurable signals:

• Cross-attention drifts away from text-relevant regions and becomes spatially unfocused or biased.
• Cross-attention becomes increasingly rigid, with a decreasing change rate in response to new tokens.
• The overall attention mass on visual tokens drops across layers as decoding proceeds.

Together, these signals form a reliable precursor of hallucination—and a direct target for intervention.

ADAPT addresses hallucination at three synergistic stages:

We extract cross-attention from the first $K$ early decoding tokens (the reliable Visually Grounded Phase) and refine them into a stable visual anchor. Each layer's attention map is scored with three complementary criteria:

• Spectral Consistency — penalizes high-frequency noise via FFT energy ratio.
• Spatial Smoothness — favors contiguous object regions over fragmented activations.
• Adaptive Focus — matches the attention concentration to the query-dependent target entropy.

The layer-wise scores are projected to fusion weights via a learned linear combination and Softmax, producing a weighted multi-layer attention map. We then apply spatial bias correction using a debiasing mask estimated from noise-image runs, which suppresses attention-sink tokens that attract spurious focus regardless of image content. The resulting anchor is query-aware, layer-selective, and corrected for positional bias.

During autoregressive decoding, we monitor cross-attention against the anchor using the Anchor-Modulated Concentration (AMC) index:

$$S_{\text{AMC}}(A_t | A_{\text{anchor}}) = \frac{\sum_i a_{i,t}^2 \cdot w_i^{\text{anchor}}}{(\sum_i a_{i,t})^2}$$

AMC jointly penalizes unfocused attention (via $a_i^2$) and spatial bias (via down-weighting regions with near-zero anchor weights). When $S_{\text{AMC}}$ falls below a threshold $\tau$, we apply sparse corrective steering:

$$\hat{a}_{i,t} = a_{i,t} + \alpha \cdot \log(w_i^{\text{anchor}} + \epsilon)$$

This injects an anchor-derived log-prior that softly re-steers attention toward visually grounded regions without imposing a hard constraint. In practice, intervention is sparse—activating primarily in the Prior-Dominated Phase where hallucinations are most likely.

We strengthen the model's intrinsic preference for visually grounded responses through preference optimization. Unlike standard DPO that contrasts at the textual level, VAG-DPO contrasts under different visual evidence conditions:

• Chosen — factual response conditioned on an anchor-enhanced image that highlights query-relevant regions.
• Rejected — hallucinatory response paired with a noise-injected image, creating a "blind" condition.

By explicitly decoupling visual grounding from linguistic priors, VAG-DPO penalizes high-confidence hallucinations and encourages generation probabilities to remain contingent on valid visual support.

ADAPT achieves state-of-the-art performance across multiple hallucination benchmarks, reducing hallucination rates by 40–60% across mainstream backbones (LLaVA-v1.5-7B/13B, Qwen2.5-VL-3B/7B).

A lightweight training-free variant (ADAPT-TF) already surpasses most prior methods, while the full ADAPT achieves the best overall results. Despite the multi-stage design, ADAPT introduces only 1.42× inference-time overhead—significantly more efficient than competing approaches (VCD: 2.03×, OPERA: 7.20×, GF-SCD: 3.96×).

ADAPT/
├── adapt/
│   ├── config.py          # Configuration class and default hyperparameters
│   ├── hook_manager.py    # Lightweight PyTorch hook registration system
│   ├── hook_logger.py     # Cross-attention recording + modification (HHI & quality modes)
│   ├── scorer.py          # Multi-criteria attention quality scoring + layer fusion
│   ├── anchor.py          # Visual anchor construction, refinement, and visualization
│   ├── inference.py       # AMC scoring and attention steering operator
│   └── utils.py           # Image I/O and tensor conversion utilities
├── eval/
│   ├── run_VE_image.py    # Convert any image to Visual Enhanced (VE) image
│   ├── test_images/       # Sample test images
│   └── results/           # Output VE images and anchor heatmaps
├── examples/
│   └── demo.py            # Single image-question inference demo
├── image/                 # Paper figures
├── requirements.txt
├── setup.py
└── README.md

ADAPT requires a working LLaVA environment. Install LLaVA first:

git clone https://github.com/haotian-liu/LLaVA.git
cd LLaVA
pip install -e .

git clone https://github.com/yao-ustc/ADAPT.git
cd ADAPT
pip install -e .

The code has been tested with LLaVA-v1.5-7B/13B and Qwen2.5-VL-3B/7B.

from adapt import ADAPTConfig, hook_logger_multi
from adapt.anchor import compute_anchor, refine_cross_attention_anchor

# Load your LLaVA model
# model = ...

cfg = ADAPTConfig()

# Register ADAPT hooks on all cross-attention layers
hook_loggers = hook_logger_multi(
    model, model.device,
    layer_indices=list(range(32)),
    modify_attention=True,       # enable attention supervision
    mode="quality",              # 'hhi' or 'quality' redistribution
    initial_phase_calls=5,       # K: early tokens for anchor
    adaptive_k1=0.4,             # intervention strength
    cfg=cfg,
)

# Generate as usual — hooks apply supervision automatically
# output = model.generate(...)

# Extract the cross-attention visual anchor
layer_maps = {}
for layer_idx, logger in hook_loggers.items():
    anchor = logger.get_anchor_map(avg_tokens=5)
    if anchor is not None:
        layer_maps[layer_idx] = anchor

fused_anchor = compute_anchor(layer_maps, cfg=cfg)

# Visualize: overlay anchor on the input image
refine_cross_attention_anchor(
    layer_maps,
    image_path="input.jpg",
    output_path="anchor_heatmap.jpg",
)

# Basic demo: generate VE image
python eval/run_VE_image.py \
    --model-path liuhaotian/llava-v1.5-7b \
    --image path/to/image.jpg \
    --question "Describe this image in detail." \
    --output-ve ve_output.jpg \
    --gpu 0

# With reduced edge penalty and anchor heatmap
python eval/run_VE_image.py \
    --model-path liuhaotian/llava-v1.5-7b \
    --image path/to/image.jpg \
    --question "What objects are in this image?" \
    --output-ve results/ve_output.jpg \
    --save-anchor-raw results/anchor.png \
    --gpu 0 \
    --boundary-penalty-strength 0.5 \
    --boundary-penalty-threshold 0.9 \
    --edge-halving-factor 0.8

ADAPT can convert any image into a Visual Enhanced (VE) image, which overlays the cross-attention anchor heatmap onto the original image. High-attention regions remain clear while low-attention regions fade to white, making the model's visual grounding instantly visible.

python eval/run_VE_image.py \
    --model-path /path/to/llava-v1.5-7b \
    --image input.jpg \
    --question "Describe this image in detail." \
    --output-ve output_ve.jpg \
    --gpu 0

If you find ADAPT useful for your research, please cite:

@inproceedings{yao2026adapt,
  title     = {ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs},
  author    = {Yao, Zhiyuan and Fu, Zheren and Zheng, Zhixiao and Li, Jiajun and Tu, Yi and Mao, Zhendong},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

This project is released under the MIT License.