Skip to content

Latest commit

 

History

History
100 lines (73 loc) · 4.82 KB

README.md

File metadata and controls

100 lines (73 loc) · 4.82 KB
Raw

Omni-RES

Python PyTorch

Omni-RES is a simple and lightweight codebase for the research of Omni-supervised referring expression segmentation, which currently supporting SimRES as base model. Later LAVT and ReLA models also will be updated.

Installation

  • Clone this repo
  • Create a conda virtual environment and activate it
conda create -n omni_res python=3.7 -y
conda activate omni_res
pip install -r requirements.txt
wget https://github.com/explosion/spacy-models/releases/download/en_vectors_web_lg-2.1.0/en_vectors_web_lg-2.1.0.tar.gz -O en_vectors_web_lg-2.1.0.tar.gz
pip install en_vectors_web_lg-2.1.0.tar.gz

Data preparation

  •  Follow the instructions of DATA_PRE_README.md to generate training data and testing data.
  •  Download the pretrained weights of backbone (vgg, darknet, cspdarknet, DResNet, etc.). Expect for DResNet, all pretrained backbones are trained on COCO 2014 train+val set while removing the images appeared in the val+test sets of RefCOCO, RefCOCO+ and RefCOCOg (nearly 6500 images). Please follow the instructions of DATA_PRE_README.md to download them.
  •  Also, we provide the necessary json files for training, the downloading url is https://anonymous.4open.science/r/omni_res_data

Training and Evaluation

  1. Config preparation. Prepare your own configs in configs, you don't need to rewrite all the contents in config every time.You can import the config as a python file to use the default configs. For example, to run 10% RefCOCO with omni-box，you can use simres_refcoco_omni.py as follows:
# your own config.py
from .common.dataset import dataset
from .common.train import train
from .common.optim import optim
from .common.models.simres import model

# Refine data path depend your own need
dataset.ann_path["refcoco"] = "./data/anns/refcoco.json"
dataset.image_path["refcoco"] = "./data/images/train2014"
dataset.mask_path["refcoco"] = "./data/masks/refcoco"
dataset.sup_ann_path["refcoco"] = "./data/anns/refcoco_0.1.json"
dataset.omni_ann_path["refcoco"] = "./data/anns/refcoco_0.9.json"
...
  1. Train the model. To start the training, you can input command as follows:
# Training 10% RefCOCO with omni-box (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_omni.py --config configs/simres_refcoco_omni.py

# Training 100% RefCOCO in fully supervised learning (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_sup.py --config configs/simres_refcoco_sup.py

# Training RefCOCO, RefCOCO+, RefCOCOg, Visual Genome with omni-box (SimRES as base model)
python -m torch.distributed.launch --nproc_per_node 4 train_vg.py --config configs/simres_refcoco_vg.py

The training logs, config.yaml and model checkpoints will be automatically saved under cfg.train.output_dir.

  1. Resume training. We support two resume training mode. You can resume from a specific checkpoint or resume from the latest checkpoint:
  • Auto resume from last.pth:
# config.py
from .common.train import train
train.auto_resume.enabled = True

Setting train.auto_resume.enabled=True, which will automatically resume from last_checkpoint.pth saved in cfg.train.output_dir.

  • Resume from a specific checkpoint
# config.py
from .common.train import train

# disable auto resume first
train.auto_resume.enabled = False

# modify the resume path
train.resume_path = "path/to/specific/checkpoint.pth"

Setting train.resume_path to the specific checkpoint.pth you want to resume from.

  1. Test the model.
python -m torch.distributed.launch --nproc_per_node 4 eval_engine.py --config configs/simres_refcoco_omni.py --eval-weights /path/to/checkpoint

License

This project is released under the Apache 2.0 license.