Omni-RES

Omni-RES is a simple and lightweight codebase for the research of Omni-supervised referring expression segmentation, which currently supporting SimRES as base model. Later LAVT and ReLA models also will be updated.

Installation

• Clone this repo
• Create a conda virtual environment and activate it

conda create -n omni_res python=3.7 -y
conda activate omni_res

• Install Pytorch following the official installation instructions
• Install mmcv following the installation guide
• Install Spacy and initialize the GloVe and install other requirements as follows:

pip install -r requirements.txt
wget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz
pip install en_vectors_web_lg-2.1.0.tar.gz

Data preparation

•  Follow the instructions of DATA_PRE_README.md to generate training data and testing data.
•  Download the pretrained weights of backbone (vgg, darknet, cspdarknet, DResNet, etc.). Expect for DResNet, all pretrained backbones are trained on COCO 2014 train+val set while removing the images appeared in the val+test sets of RefCOCO, RefCOCO+ and RefCOCOg (nearly 6500 images). Please follow the instructions of DATA_PRE_README.md to download them.
•  Also, we provide the necessary json files for training, the downloading url is https://anonymous.4open.science/r/omni_res_data

Training and Evaluation

1. Config preparation. Prepare your own configs in configs, you don't need to rewrite all the contents in config every time.You can import the config as a python file to use the default configs. For example, to run 10% RefCOCO with omni-box，you can use simres_refcoco_omni.py as follows:

# your own config.py
from .common.dataset import dataset
from .common.train import train
from .common.optim import optim
from .common.models.simres import model

# Refine data path depend your own need
dataset.ann_path["refcoco"] = "./data/anns/refcoco.json"
dataset.image_path["refcoco"] = "./data/images/train2014"
dataset.mask_path["refcoco"] = "./data/masks/refcoco"
dataset.sup_ann_path["refcoco"] = "./data/anns/refcoco_0.1.json"
dataset.omni_ann_path["refcoco"] = "./data/anns/refcoco_0.9.json"
...

2. Train the model. To start the training, you can input command as follows:

# Training 10% RefCOCO with omni-box (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_omni.py --config configs/simres_refcoco_omni.py

# Training 100% RefCOCO in fully supervised learning (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_sup.py --config configs/simres_refcoco_sup.py

# Training RefCOCO, RefCOCO+, RefCOCOg, Visual Genome with omni-box (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_vg.py --config configs/simres_refcoco_vg.py

The training logs, config.yaml and model checkpoints will be automatically saved under cfg.train.output_dir.

3. Resume training. We support two resume training mode. You can resume from a specific checkpoint or resume from the latest checkpoint:

• Auto resume from last.pth:

# config.py
from .common.train import train
train.auto_resume.enabled = True

Setting train.auto_resume.enabled=True, which will automatically resume from last_checkpoint.pth saved in cfg.train.output_dir.

• Resume from a specific checkpoint

# config.py
from .common.train import train

# disable auto resume first
train.auto_resume.enabled = False

# modify the resume path
train.resume_path = "path/to/specific/checkpoint.pth"

Setting train.resume_path to the specific checkpoint.pth you want to resume from.

4. Test the model.

python -m torch.distributed.launch --nproc_per_node 4 eval_engine.py --config configs/simres_refcoco_omni.py --eval-weights /path/to/checkpoint

License

This project is released under the Apache 2.0 license.