The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?
This repository contains the source code for the paper.
The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models? [Paper]
Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, Stephen Gould
Large vision-language models (LVLMs), designed to interpret and respond to human instructions, occasionally generate hallucinated or harmful content due to inappropriate instructions. This study uses linear probing to shed light on the hidden knowledge at the output layer of LVLMs. We demonstrate that the logit distributions of the first tokens contain sufficient information to determine whether to respond to the instructions, including recognizing unanswerable visual questions, defending against multi-modal jailbreaking attack, and identifying deceptive questions. Such hidden knowledge is gradually lost in logits of subsequent tokens during response generation. Then, we illustrate a simple decoding strategy at the generation of the first token, effectively improving the generated content. In experiments, we find a few interesting insights: First, the CLIP model already contains a strong signal for solving these tasks, indicating potential bias in the existing datasets. Second, we observe performance improvement by utilizing the first logit distributions on three additional tasks, including indicting uncertainty in math solving, mitigating hallucination, and image classification. Last, with the same training data, simply finetuning LVLMs improve models' performance but is still inferior to linear probing on these tasks.
git clone https://github.com/Qinyu-Allen-Zhao/LVLM-LP.git
We evaluate the linear probing method on six different tasks. For a specific task, you need to (i) prepare the dataset, (ii) prepare the model, (iii) run the model on the dataset to get logits of the first tokens, and last, (iv) evaluate the performance of linear probing.
Don't worry. We are trying our best to make the process very simple for you.
After downloading and unzipping data, please modify dataset paths here.
-
Task 1. Identify unanswerable visual questions: Download images and annotations of the train and validation sets of VizWiz.
-
Task 2. Defense against jailbreak attack: We use MM-SafetyBench and generated safe image-query pairs via the same data generation process of theirs. Our generated data is provided on GoogleDrive.
-
Task 3. Identify deceptive questions: When we conducted the experiment, the dataset MAD-Bench had not been released yet. We generated the data via the process described in the paper and augmented the dataset by adding normal questions. Download the val2017 set of COCO. The annotations have been included in this repo.
-
Task 4. Uncertainty in math solving: We use the testmini set of MathVista. It will be automatically downloaded via HuggingFace. You don't need to do anything at this stage.
-
Task 5. Mitigate hallucination: Download the train2014 and val2014 sets of COCO. The annotations have been included in this repo.
-
Task 6. Image classification: Download the train and val sets of ImageNet-1k.
Please prepare the models that you want to use, as illustrated in their original repos. We didn't modify the main structure of their models.
- Clone their repos first. LLaVA, MiniGPT-4, InstructBLIP, mPLUG-Owl, LLaMA-Adapter, MMGPT. *LLaVA might be an easy start.
- Download model checkpoints as required;
- Set the model folder in the model files under model.
OK, life goes very easy at this stage. We provide you all scripts for running models on each task to get logits of the first tokens. Please find them here.
For example, run this command (modify it based on your gpu numbers)
CUDA_VISIBLE_DEVICES=0,1,2,3 bash ./scripts/VizWiz/run_LLaVA_7B.sh
Step 3 is to generate and store logits of the first tokens. In this step, we will run linear probing (a simple classifer actually) and evaluate its performance.
Please refer to those eval.ipynb files.
Finetuned with training data used in the 6 tasks: model. We finetune a pretrained LLaVA v1.5 7B model for 15 epochs and save checkpoint after each epoch.
Retrained with original instruction-tuning data and our training data: model. We train the model for one epoch as the authors did.
After you download the model checkpoints, please modify model_path in finetune_all.sh and retrain_all.sh, to run the two models.
-
Upload data generated in our study
-
Write a step-by-step instruction in the README file
-
Upload the checkpoints of finetuned LLaVA
-
Upload evaluation codes for all six tasks
If you use our codebase or our results in your research, please cite our work:
@article{zhao2024first,
title={The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?},
author={Zhao, Qinyu and Xu, Ming and Gupta, Kartik and Asthana, Akshay and Zheng, Liang and Gould, Stephen},
journal={arXiv preprint arXiv:2403.09037},
year={2024}
}