CLOVER: Cost-effective Instruction Learning for Pathology Vision and Language Analysis

---

📅 Latest News

[2025/06] 🎉 Paper Published in Nature Computational Science! Our paper "Cost-effective Instruction Learning for Pathology Vision and Language Analysis" has been officially published in Nature Computational Science.

[2024/07] 📝 arXiv Preprint Available! Our paper "Cost-effective Instruction Learning for Pathology Vision and Language Analysis" is now available on arXiv.

---

📖 Overview

CLOVER is a cost-effective instruction learning framework designed for conversational pathology analysis. It addresses the challenges of deploying vision-language models in clinical settings by providing an efficient training approach that requires minimal computational resources while maintaining high performance.

🎯 Key Features

• 🔄 Two-Stage Training: Vision-language alignment + instruction fine-tuning
• 💰 Cost-Effective: Uses GPT-3.5 for instruction generation only costs $8
• ⚡ Lightweight: Only trains a small module while freezing LLM parameters
• 🏥 Domain-Specific: Optimized for pathology analysis with specialized instructions
• 📊 High Performance: Outperforms baselines with 37x more training parameters

---

🧠 Architecture

A schematic illustration of CLOVER. a The workflow for instruction generation. b The distribution of covered body parts or cancer types. c The distribution of question and answer sentence lengths. d The workflow of CLOVER.

Training Stages

---

⚙️ Installation

Prerequisites

• Python 3.8+
• CUDA-compatible GPU

Step-by-Step Installation

1. Create Conda Environment

   conda create -n clover python=3.9
   conda activate clover

2. Clone Repository

   git clone https://github.com/JLINEkai/CLOVER.git
   cd CLOVER

3. Install Dependencies

   pip install -r requirements.txt

---

🔧 Usage

Training

Stage 1: Vision-Language Alignment

python train_blip2qformer.py

Stage 2: Instruction Fine-tuning

python -m torch.distributed.run --nproc_per_node=1 train.py

Note: You can choose the large language model (LLM) in lavis/projects/blip2/train/pretrain_stage2.yaml. We provide support for FlanT5XL and Vicuna 7B.

Inference

python -m torch.distributed.run --nproc_per_node=1 evaluate.py --cfg-path lavis/projects/blip2/eval/vqav2_zeroshot_flant5xl_eval.yaml

---

📊 Results & Performance

Benchmark Results

CLOVER demonstrates superior performance on pathology vision-language tasks.

Quantitative Comparison

Comparison with SOTA methods on PathVQA dataset.

Comparison with prior SOTA methods on QUILT-VQA dataset.

Qualitative Examples

Qualitative comparisons of visual question answering.

---

📁 Data

Required Datasets

1. Quilt-1M Dataset (Stage 1)

   • Download from Google Drive
   • Alternative: Zenodo

2. CLOVER Instructions (Stage 2)

   • Will be released soon
   • You can generate your own using our prompts in generate_instructions.py

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🙏 Acknowledgments

• Based on BLIP-2 framework

• Inspired by recent advances in vision-language models

---

📚 Citation

If you find this work useful, please cite our paper:

@article{chen2025cost,
  title={Cost-effective instruction learning for pathology vision and language analysis},
  author={Chen, K. and Liu, M. and Yan, F. and others},
  journal={Nature Computational Science},
  year={2025},
  doi={10.1038/s43588-025-00818-5}
}

---

_{Made with ❤️ for the pathology research community}