This repository includes the code for paper Context-aware zero-shot recognition.
The code is highly based on Detectron.pytorch. Thanks for the effort from original author.
- Pytorch 1.0
- Python 2 and python 3
- torchtext
- nltk
- pycocotools
bash data/vg/download.sh
(In python 2)
python lib/datasets/vg/convert_from_bansal.py
cd lib
bash make.sh
cd ..
python tools/download_imagenet_weights.py
Download pretrained model from link.
run scripts in scripts/reproduce
bash scripts/train.sh
In this case, --cfg defines the general training test configuration, like roi align size, backbone bone etc.
--auto_resume enables it to resume from the latest snapshot, this has to be used with --id together.
--set overrides the items in the configuration files. Here we manually set NUM_CLASSES BASE_LR and MODEL.TAGGING.
--id allow us to identify where we save our model much easier.
iter_reason_new.yaml is borrowed from xinlei's iterative reasoning paper.
bash scripts/train_we.sh
This append MODEL.WORD_EMBEDDING_REGU True
in '--set'.
Normally, the last fc layer of fast rcnn head is 2048x601, where here this layer is replaced by two linear layer 2048x300 and 300x601. The weight of 300x601 layer is the word embedding of all the classes.
bash train_rel.sh (bash train_we_rel.sh)
bash scripts/test/we_infer.sh
bash scripts/test/we_noinfer.sh
bash scripts/test/conse_infer.sh
bash scripts/test/conse_noinfer.sh
Download zsl-pth-gcn to ./externals
.
(In python 2)
python lib/zsl-gcn/convert_gcn.py --weight_folder ./Outputs/rel_ft/ft_gt_relt_geo_sc
bash scripts/test/gcn_infer.sh
bash scripts/test/gcn_noinfer.sh
(In python 2)
python lib/zsl-gcn/convert_sync.py --weight_folder ./Outputs/iter_reason_new/irn_vg_gt_sc_step --dataset vgbansal --to_weight Outputs/rel_ft/ft_gt_relt_geo_sc_step/ckpt/model_final.pth
cd ./external/edges
get_proposals.m
python to_raw_proposals.py
cd ../../data/vg/bansal/
python generate_proposals.py
or download from link, and put it under ./data/vg/bansal
bash scripts/test/detection_gcn.sh
bash scripts/test/detection_sync.sh