Skip to content

zizhaozhang/distill2

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Text-guided Neural Network Training for Image Recognition in Natural Scenes and Medicine

This repo contains the source code and prepared datasets of the paper.

Preparation

Prerequisites

Follow the following steps to prepare the running environment.

  1. Clone and enter into the repo.
  2. Prepare folders for dataset and checkpoints.
$ mkdir dataset
$ mkdir checkpoints
  1. Create a conda virtual environment and install dependencies
$ conda create --name distill2 python=3.6
$ conda activate distill2
$ pip install -r requirements.txt

Datasets

The method is tested on four datasets: Vgnome, BICIDR(This version is deprecated. Please goto this repo to find the newest version.), ChestXray, and COCO2014. COCO2014 can be organized following the offical settings, the other three datasets are prepared by us. For convenient experiments, download them through the given links and save them inside the datasets/.

Text embedding using Glove

We have created embeddings for the datasets (only COCO and Vgnome use them), which can be accessed from coco_glove and vgnome_glove.

  1. Download these two embedded glove pickle files and copy to dataset/
  2. (Optional) The following command line creates initial embedding matrix.
python data/create_initial_embedding.py

The code script requires vocabularies of the dataset (see the source code for more details), which we have provided in data/vocab_corpus. In addition, glove.6B.300d.txt need to be placed under dataset/.

Model Training

The following instruction works through experiments on COCO. All other launch scripts on the other datasets can be found at scripts/

  • Train TandemNet2
    device=0 sh scripts/coco/train_tandemnet2.sh
    
  • Train TandemNet
    device=0 sh scripts/coco/train_tandemnet.sh
    
  • (Optional) Train ResNet101
    device=0 sh scripts/coco/train_cnn.sh
    

Model Testing

  • Test TandemNet2
    device=0 sh scripts/coco/test_tandemnet2.sh
    
  • Test TandemNet
    device=0 sh scripts/coco/test_tandemnet.sh
    
  • (Optional) Test ResNet101
    device=0 sh scripts/coco/test_cnn.sh
    

Citation

Please consider to cite our papers

@article{zhang2019text,
    title={Text-guided Neural Network Training for Image Recognition in Natural Scenes and Medicine},
    author={Zhang, Zizhao and Chen, Pingjun and Shi, Xiaoshuang and Yang, Lin},
    journal={IEEE transactions on pattern analysis and machine intelligence},
    year={2019},
    publisher={IEEE}
}
@inproceedings{Zhang2017TandemNet,
    title={TandemNet: Distilling Knowledge from Medical Images Using Diagnostic Reports as Optional Semantic References},
    author={Zhang, Zizhao and Chen, Pingjun and Sapkota, Manish and Yang, Lin},
    booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)},
    year={2017}
}

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published