HiProbe-VAD: Video Anomaly Detection via Hidden States Probing

This repository contains the official implementation of the paper "HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs" (ACM MM 2025).

HiProbe-VAD is a novel, tuning-free framework that leverages the "Intermediate Layer Information-rich Phenomenon" in Multimodal Large Language Models (MLLMs). By probing intermediate hidden states, it achieves state-of-the-art anomaly detection performance without requiring fine-tuning of the MLLM.

Key Features

• Tuning-Free: Leverages pre-trained MLLMs (InternVL2.5, Qwen2.5-VL, etc.) without parameter updates.
• Dynamic Layer Saliency Probing (DLSP): Automatically identifies the optimal intermediate layer for anomaly detection using a few-shot subset.
• Lightweight Scorer: Efficient logistic regression classifier trained on extracted hidden states.
• Explainable VAD: Generates detailed text descriptions for detected anomalies.

Installation

1. Clone the repository:

   git clone https://github.com/your-username/hiprobe-vad.git
   cd hiprobe-vad

2. Create a virtual environment (optional but recommended):

   conda create -n hiprobe python=3.10
   conda activate hiprobe

3. Install dependencies:

   pip install -r requirements.txt
   pip install -e .

Usage

1. Prepare Data

Create a manifest file (JSONL or CSV) listing your videos and their labels. Example manifest.jsonl:

{"video": "/path/to/videos/normal_01.mp4", "label": "N"}
{"video": "/path/to/videos/anomaly_01.mp4", "label": "A"}

Update the data section in configs/hiprobe_example.yaml to point to your manifest.

2. Extract Hidden States

Extract hidden states from the MLLM. This step processes the videos and saves feature vectors.

hiprobe-extract --config configs/hiprobe_example.yaml

3. Run DLSP (Layer Selection)

Identify the optimal layer using the few-shot training subset.

hiprobe-dlsp \
  --config configs/hiprobe_example.yaml \
  --activations-dir ./outputs/activations \
  --output ./outputs/dlsp_metrics.json

4. Train Anomaly Scorer

Train the lightweight classifier on the selected optimal layer (e.g., layer 20).

hiprobe-train \
  --config configs/hiprobe_example.yaml \
  --activations-dir ./outputs/activations \
  --layer 20 \
  --output-model ./outputs/scorer.joblib \
  --output-stats ./outputs/scorer_stats.json

5. Inference

Run inference on a new video to detect anomalies and optionally generate explanations.

hiprobe-infer \
  --config configs/hiprobe_example.yaml \
  --video /path/to/test_video.mp4 \
  --model ./outputs/scorer.joblib \
  --stats ./outputs/scorer_stats.json \
  --layer 20 \
  --explain

Configuration

The system is configured via YAML files (e.g., configs/hiprobe_example.yaml). Key parameters include:

• model: Backend (internvl, qwen2.5vl, etc.), model path, and keyframe sampling settings (keyframes, segment_frames).
• data: Dataset paths and label definitions.
• dlsp: Number of bins for entropy calculation.
• scorer: Logistic regression hyperparameters.
• localization: Gaussian smoothing ($\sigma$) and adaptive threshold ($\kappa$) parameters.

Citation

If you find this work useful, please cite our paper:

@inproceedings{cai2025hiprobe,
  title={HiProbe-VAD: Video Anomaly Detection via Hidden States Probing in Tuning-Free Multimodal LLMs},
  author={Cai, Zhaolin and Li, Fan and Zheng, Ziwei and Qin, Yanjun},
  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
  year={2025},
  doi={10.1145/3746027.3755575}
}

License

This project is licensed under the MIT License.