HalLoc: Token-level Hallucination Localization for Vision-Language Models

Official codebase and dataset for the CVPR 2025 paper: "HalLoc: Token-level Hallucination Localization for Vision-Language Models" Eunkyu Park*, Minyeong Kim*, Gunhee Kim, Seoul National University (* equal contribution)

Paper PDF | Project Page | HuggingFace Dataset

---

Overview

HalLoc introduces a benchmark and detection model for token-level hallucination localization in vision-language model (VLM) outputs. Unlike prior works, HalLoc supports:

• Fine-grained hallucination detection across object, attribute, relationship, and scene categories
• Probabilistic outputs, enabling nuanced interpretation via calibrated confidence scores
• Real-time detection, designed for plug-and-play integration with VLMs

The dataset includes 155K token-level annotated samples across Visual Question Answering, Instruction Following, and Image Captioning tasks.

We also provide a lightweight detection model, HalLocalizer, built on VisualBERT that operates on VLM hidden-state embeddings + images.

---

Repository Structure

cvpr25_halloc/
├── train.py                               # Training entry point (Hydra + PyTorch Lightning)
├── config/                                # Hydra configuration files
│   ├── train.yaml                         #   root config
│   ├── datamodule/                        #   data loading configs
│   ├── dataset/                           #   dataset configs
│   ├── model/                             #   model architecture configs
│   ├── module/                            #   lightning module configs
│   ├── optimizer/                         #   optimizer configs (AdamW)
│   ├── scheduler/                         #   LR scheduler configs (cosine, multi-step)
│   ├── loss/                              #   loss function configs (cross-entropy)
│   ├── metric/                            #   metric configs (classification)
│   ├── callback/                          #   callback configs
│   ├── trainer/                           #   trainer configs
│   └── experiment/                        #   experiment presets
│
├── src/                                   # Core library
│   ├── model/                             #   HalLocalizer model definitions
│   │   ├── halloc.py                      #     embedding-based model (VLM hidden states)
│   │   └── halloc_text.py                 #     text-based model (tokenized input)
│   ├── module/                            #   PyTorch Lightning modules
│   │   ├── loss/                          #     loss functions
│   │   ├── metric/                        #     evaluation metrics
│   │   └── optimizer/                     #     optimizer + scheduler setup
│   ├── datamodule/                        #   data modules + datasets
│   │   └── dataset/                       #     dataset implementations
│   ├── message/                           #   inter-component messaging
│   └── utils/                             #   logging utilities
│
└── scripts/                               # Pipeline scripts
    ├── extract/                           #   Step 1: Extract VLM embeddings
    │   ├── extract_vlm_embeddings_llava.py
    │   ├── extract_vlm_embeddings_internvl.py
    │   ├── extract_vlm_embeddings_iblip.py
    │   ├── extract_vlm_embeddings_minigpt4.py
    │   ├── extract_vlm_embeddings_text.py
    │   └── extract_vlm_embeddings_text_blur.py
    │
    ├── postprocess/                       #   Step 2: Align token indices
    │   ├── postprocess_llava.py
    │   ├── postprocess_internvl.py
    │   ├── postprocess_iblip.py
    │   ├── postprocess_minigpt4.py
    │   └── postprocess_text.py
    │
    ├── evaluate/                          #   Step 4: Evaluate + find thresholds
    │   ├── evaluate_single.py
    │   ├── evaluate_single_text.py
    │   ├── calculate_optimal_threshold.py
    │   └── calculate_optimal_threshold_text.py
    │
    └── calibration/                       #   Step 5: Calibration analysis
        ├── calculate_calibration_error_halloc_ece.py
        ├── calculate_calibration_error_halloc_ace.py
        ├── calculate_calibration_error_internvl_ece.py
        ├── calculate_calibration_error_internvl_ace.py
        └── calculate_logprob_internvl.py

---

Requirements

• Python >= 3.8
• PyTorch >= 1.13
• PyTorch Lightning
• Hydra (hydra-core, hydra-colorlog)
• HuggingFace Transformers
• Accelerate
• fire, loguru, scikit-learn, Pillow

For VLM-specific extraction scripts, you also need the corresponding VLM libraries:

• LLaVA: liuhaotian/llava-v1.5-7b
• InternVL: OpenGVLab/InternVL2-8B
• InstructBLIP / MiniGPT-4: respective repositories

---

Dataset

The HalLoc dataset contains 155K token-level annotated samples with hallucination labels across five categories: object, attribute, relationship, scene, and other.

Download link: Coming soon (HuggingFace / Google Drive)

The annotation JSON files follow this structure:

{
  "id": "sample_id",
  "image_id": "image_id",
  "prompt": "<image> question text",
  "hallucinated_text": "model response text",
  "tokenized_text": ["token1", "token2", ...],
  "annotations": {
    "object": [{"entity": {"name": "...", "char_index": "start:end", "token_index": "start:end"}}],
    "attribute": [...],
    "relationship": [...],
    "scene": [...],
    "other": [...]
  }
}

Images come from Visual Genome (VG_100K / VG_100K_2) and MS-COCO (train2014 / val2014).

---

Pipeline

The full workflow has five stages:

Step 1: Postprocess annotations

Align character-level hallucination annotations to VLM-specific tokenizations:

python scripts/postprocess/postprocess_llava.py \
    --input_path data/halloc_train.json

Step 2: Extract VLM embeddings

Run a VLM in forward mode to extract hidden-state embeddings for each sample. Uses HuggingFace Accelerate for multi-GPU:

accelerate launch scripts/extract/extract_vlm_embeddings_llava.py \
    --data_path data/halloc_train_llava_postprocessed.json \
    --image_dir /path/to/images \
    --save_dir data/train/vlm_embeddings/llava \
    --batch_size 8

Step 3: Train HalLocalizer

Train the hallucination detection model using Hydra configs:

python train.py \
    experiment.name=halloc_llava \
    experiment.work_dir=./outputs

To use the text-based model variant:

python train.py \
    datamodule=halloc_text \
    module=halloc_text \
    model=halloc_text \
    experiment.name=halloc_text

Override any config value via the command line (Hydra syntax). See config/ for all options.

Step 4: Evaluate

Find optimal per-category thresholds on validation set, then evaluate:

python scripts/evaluate/calculate_optimal_threshold.py \
    --checkpoint_path outputs/halloc_llava/best.ckpt \
    --data_dir data/val/vlm_embeddings/llava

python scripts/evaluate/evaluate_single.py \
    --threshold_path evaluation/thresholds/thresholds.json \
    --checkpoint_path outputs/halloc_llava/best.ckpt \
    --data_dir data/val/vlm_embeddings/llava

Step 5: Calibration analysis (optional)

Compute Expected Calibration Error (ECE) and Adaptive Calibration Error (ACE) with temperature scaling:

python scripts/calibration/calculate_calibration_error_halloc_ece.py --subset vqa
python scripts/calibration/calculate_calibration_error_internvl_ece.py --subset vqa

---

Citation

If you find this work helpful, please consider citing:

@inproceedings{park2025halloc,
  title={HalLoc: Token-level Hallucination Localization for Vision-Language Models},
  author={Park, Eunkyu and Kim, Minyeong and Kim, Gunhee},
  booktitle={CVPR},
  year={2025}
}

Acknowledgements

This work was supported by Seoul National University and grants from MSIT/IITP (South Korea).

---

Contact

For questions or collaborations, feel free to open an issue or email us at: eunkyu.park@vision.snu.ac.kr