-
Notifications
You must be signed in to change notification settings - Fork 12
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
2 changed files
with
59 additions
and
186 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,196 +1,71 @@ | ||
| # RefineDet in PyTorch | ||
| This is a PyTorch implementation of [Single-Shot Refinement Neural Network for Object Detection](https://arxiv.org/abs/1711.06897) which is a 2018CVPR paper written by Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei and Stan Z. Li. The official and original Caffe code can be found [here](https://github.com/sfzhang15/RefineDet). | ||
| This is a PyTorch implementation of [Single-Shot Refinement Neural Network for Object Detection](https://arxiv.org/abs/1711.06897) that is a work by Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei and Stan Z. Li in CVPR2018. The official and original Caffe code can be found [here](https://github.com/sfzhang15/RefineDet). | ||
| This implementation mainly refers the official RefineDet in Caffe [sfzhang15/RefineDet](https://github.com/sfzhang15/RefineDet) and a PyTorch implementation of SSD [amdegroot/ssd.pytorch](https://github.com/amdegroot/ssd.pytorch). Its sublibrary(*libs/dataset/datasets*) to process datasets is obtained from [jwyang/faster-rcnn.pytorch](https://github.com/jwyang/faster-rcnn.pytorch) with minor modification. | ||
|
|
||
| This implementation mainly refers the official RefineDet in Caffe [sfzhang15/RefineDet](https://github.com/sfzhang15/RefineDet) and a PyTorch implementation of SSD [amdegroot/ssd.pytorch](https://github.com/amdegroot/ssd.pytorch). Its sublibrary(*pytorch_refinedet/libs/dataset/datasets*) to process datasets is obtained from [jwyang/faster-rcnn.pytorch](https://github.com/jwyang/faster-rcnn.pytorch) with minor modification. | ||
|
|
||
| ## Performance | ||
|
|
||
| ## Repuisites | ||
| * Python 2.7 | ||
| * PyTorch 0.3.1 | ||
| * CUDA 8.0 or higher | ||
| ## Data Preparation | ||
| * **PASCAL_VOC 07+12**: You can follow the instructions in [py-faster-rcnn](https://github.com/rbgirshick/py-faster-rcnn#beyond-the-demo-installation-for-training-and-testing-models) or [jwyang/faster-rcnn.pytorch](https://github.com/jwyang/faster-rcnn.pytorch) to prepare VOC datasets, i.e., creating softlinks or putting the data in the folder data/ | ||
| * **COCO**: You can use COCO2014 to train your model by the same setting as PASCAL_VOC07+12. | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
| ## Data Preparation | ||
| * **PASCAL_VOC 07+12**: You can follow the instructions in [py-faster-rcnn](https://github.com/rbgirshick/py-faster-rcnn#beyond-the-demo-installation-for-training-and-testing-models) or [jwyang/faster-rcnn.pytorch](https://github.com/jwyang/faster-rcnn.pytorch) to prepare VOC datasets, i.e., putting the data or creating soft links in the folder data/. | ||
|
|
||
|
|
||
|
|
||
| ### Table of Contents | ||
| - <a href='#installation'>Installation</a> | ||
| - <a href='#datasets'>Datasets</a> | ||
| - <a href='#training-ssd'>Train</a> | ||
| - <a href='#evaluation'>Evaluate</a> | ||
| - <a href='#performance'>Performance</a> | ||
| - <a href='#demos'>Demos</a> | ||
| - <a href='#todo'>Future Work</a> | ||
| - <a href='#references'>Reference</a> | ||
|
|
||
| | ||
| | ||
| | ||
| | ||
|
|
||
| ## Installation | ||
| - Install [PyTorch](http://pytorch.org/) by selecting your environment on the website and running the appropriate command. | ||
| - Clone this repository. | ||
| * Note: We currently only support Python 3+. | ||
| - Then download the dataset by following the [instructions](#datasets) below. | ||
| - We now support [Visdom](https://github.com/facebookresearch/visdom) for real-time loss visualization during training! | ||
| * To use Visdom in the browser: | ||
| ```Shell | ||
| # First install Python server and client | ||
| pip install visdom | ||
| # Start the server (probably in a screen or tmux) | ||
| python -m visdom.server | ||
| ``` | ||
| * Then (during training) navigate to http://localhost:8097/ (see the Train section below for training details). | ||
| - Note: For training, we currently support [VOC](http://host.robots.ox.ac.uk/pascal/VOC/) and [COCO](http://mscoco.org/), and aim to add [ImageNet](http://www.image-net.org/) support soon. | ||
|
|
||
| ## Datasets | ||
| To make things easy, we provide bash scripts to handle the dataset downloads and setup for you. We also provide simple dataset loaders that inherit `torch.utils.data.Dataset`, making them fully compatible with the `torchvision.datasets` [API](http://pytorch.org/docs/torchvision/datasets.html). | ||
|
|
||
|
|
||
| ### COCO | ||
| Microsoft COCO: Common Objects in Context | ||
|
|
||
| ##### Download COCO 2014 | ||
| ```Shell | ||
| # specify a directory for dataset to be downloaded into, else default is ~/data/ | ||
| sh data/scripts/COCO2014.sh | ||
| ``` | ||
|
|
||
| ### VOC Dataset | ||
| PASCAL VOC: Visual Object Classes | ||
|
|
||
| ##### Download VOC2007 trainval & test | ||
| ```Shell | ||
| # specify a directory for dataset to be downloaded into, else default is ~/data/ | ||
| sh data/scripts/VOC2007.sh # <directory> | ||
| ``` | ||
|
|
||
| ##### Download VOC2012 trainval | ||
| ```Shell | ||
| # specify a directory for dataset to be downloaded into, else default is ~/data/ | ||
| sh data/scripts/VOC2012.sh # <directory> | ||
| ``` | ||
| * **COCO**: You can use COCO2014 to train your model by the same setting as PASCAL_VOC07+12. | ||
|
|
||
| ## Training SSD | ||
| - First download the fc-reduced [VGG-16](https://arxiv.org/abs/1409.1556) PyTorch base network weights at: https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth | ||
| - By default, we assume you have downloaded the file in the `ssd.pytorch/weights` dir: | ||
| The directory trees in data/ in my projects are as following: | ||
| ├── coco -> /root/dataset/coco | ||
| ├── VOCdevkit2007 | ||
| │ └── VOC2007 -> /root/dataset/voc/VOCdevkit/VOC2007 | ||
| └── VOCdevkit2012 | ||
| └── VOC2012 -> /root/dataset/voc/VOCdevkit/VOC2012 | ||
|
|
||
| ## Train | ||
| You can train a RefineDet detector with [VGG16](https://arxiv.org/abs/1409.1556) or [ResNet101](https://arxiv.org/abs/1512.03385) as a base network. The pretrained models can be downloaded from [vgg16_reducedfc.pth](https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth) and [resnet101.pth](https://download.pytorch.org/models/resnet101-5d3b4d8f.pth). | ||
| By default, the directories for VGG16 weights is assumed as following: | ||
| ```Shell | ||
| mkdir weights | ||
| cd weights | ||
| mkdir -p weights/vgg16 | ||
| cd weights/vgg16 | ||
| wget https://s3.amazonaws.com/amdegroot-models/vgg16_reducedfc.pth | ||
| ``` | ||
|
|
||
| - To train SSD using the train script simply specify the parameters listed in `train.py` as a flag or manually change them. | ||
|
|
||
| For ResNet101: | ||
| ```Shell | ||
| python train.py | ||
| mkdir -p weights/resnet101 | ||
| cd weights/resnet101 | ||
| wget https://download.pytorch.org/models/resnet101-5d3b4d8f.pth -O resnet101.pth | ||
| ``` | ||
|
|
||
| - Note: | ||
| * For training, an NVIDIA GPU is strongly recommended for speed. | ||
| * For instructions on Visdom usage/installation, see the <a href='#installation'>Installation</a> section. | ||
| * You can pick-up training from a checkpoint by specifying the path as one of the training parameters (again, see `train.py` for options) | ||
|
|
||
| ## Evaluation | ||
| To evaluate a trained network: | ||
|
|
||
| ```Shell | ||
| python eval.py | ||
| To train a RefineDet model with PASCAL VOC07+12, run as: | ||
| ``` | ||
| python train_refinedet.py --dataset voc --input_size 320 --network vgg16 --basenet vgg16_reducedfc.pth --save_folder "weights/vgg16" | ||
| ``` | ||
| Change dataset to "coco", if you want to use COCO2014. | ||
|
|
||
| You can specify the parameters listed in the `eval.py` file by flagging them or manually changing them. | ||
|
|
||
|
|
||
| <img align="left" src= "https://github.com/amdegroot/ssd.pytorch/blob/master/doc/detection_examples.png"> | ||
| ## Evaluation | ||
| To evaluate a trained model, run as: | ||
| python eval_refinedet.py --input_size 512 --dataset voc --network vgg16 --model_path your/weights/path | ||
|
|
||
| ## Performance | ||
| The project trys to reproduce the performance of RefineDet in Caffe, but there are some gaps. | ||
| If you have any suggetion to promote this reproduction, please leave a message in the issues. | ||
| 1). PASCAL VOC07+12 (Train/Test: VOC07+12/VOC07) | ||
| Method |Backbone | Input Size | mAP | ||
| ------|-----|-------|-----| | ||
| SSD | VGG16| 300 x 300 | 77.2 | ||
| SSD | VGG16| 512 x 512 | 79.8 | ||
| RefineDet(Official)| VGG16| 320 x 320 | 80.0 | ||
| RefineDet(Official)| VGG16| 512 x 512 | 81.8 | ||
| **RefineDet(Our)**| VGG16| 320 x 320 | 78.9 | ||
| **RefineDet(Our)**| VGG16| 512 x 512 | 79.8 | ||
|
|
||
| #### VOC2007 Test | ||
|
|
||
| ##### mAP | ||
| ## Demo | ||
| You can run demo/demo.ipynb with jupyter notebook to visualize detetion results. | ||
|
|
||
| | Original | Converted weiliu89 weights | From scratch w/o data aug | From scratch w/ data aug | | ||
| |:-:|:-:|:-:|:-:| | ||
| | 77.2 % | 77.26 % | 58.12% | 77.43 % | | ||
|
|
||
| ##### FPS | ||
| **GTX 1060:** ~45.45 FPS | ||
|
|
||
| ## Demos | ||
|
|
||
| ### Use a pre-trained SSD network for detection | ||
|
|
||
| #### Download a pre-trained network | ||
| - We are trying to provide PyTorch `state_dicts` (dict of weight tensors) of the latest SSD model definitions trained on different datasets. | ||
| - Currently, we provide the following PyTorch models: | ||
| * SSD300 trained on VOC0712 (newest PyTorch weights) | ||
| - https://s3.amazonaws.com/amdegroot-models/ssd300_mAP_77.43_v2.pth | ||
| * SSD300 trained on VOC0712 (original Caffe weights) | ||
| - https://s3.amazonaws.com/amdegroot-models/ssd_300_VOC0712.pth | ||
| - Our goal is to reproduce this table from the [original paper](http://arxiv.org/abs/1512.02325) | ||
| <p align="left"> | ||
| <img src="http://www.cs.unc.edu/~wliu/papers/ssd_results.png" alt="SSD results on multiple datasets" width="800px"></p> | ||
|
|
||
| ### Try the demo notebook | ||
| - Make sure you have [jupyter notebook](http://jupyter.readthedocs.io/en/latest/install.html) installed. | ||
| - Two alternatives for installing jupyter notebook: | ||
| 1. If you installed PyTorch with [conda](https://www.continuum.io/downloads) (recommended), then you should already have it. (Just navigate to the ssd.pytorch cloned repo and run): | ||
| `jupyter notebook` | ||
|
|
||
| 2. If using [pip](https://pypi.python.org/pypi/pip): | ||
|
|
||
| ```Shell | ||
| # make sure pip is upgraded | ||
| pip3 install --upgrade pip | ||
| # install jupyter notebook | ||
| pip install jupyter | ||
| # Run this inside ssd.pytorch | ||
| jupyter notebook | ||
| ``` | ||
|
|
||
| - Now navigate to `demo/demo.ipynb` at http://localhost:8888 (by default) and have at it! | ||
|
|
||
| ### Try the webcam demo | ||
| - Works on CPU (may have to tweak `cv2.waitkey` for optimal fps) or on an NVIDIA GPU | ||
| - This demo currently requires opencv2+ w/ python bindings and an onboard webcam | ||
| * You can change the default webcam in `demo/live.py` | ||
| - Install the [imutils](https://github.com/jrosebr1/imutils) package to leverage multi-threading on CPU: | ||
| * `pip install imutils` | ||
| - Running `python -m demo.live` opens the webcam and begins detecting! | ||
|
|
||
| ## TODO | ||
| We have accumulated the following to-do list, which we hope to complete in the near future | ||
| - Still to come: | ||
| * [x] Support for the MS COCO dataset | ||
| * [ ] Support for SSD512 training and testing | ||
| * [ ] Support for training on custom datasets | ||
|
|
||
| ## Authors | ||
|
|
||
| * [**Max deGroot**](https://github.com/amdegroot) | ||
| * [**Ellis Brown**](http://github.com/ellisbrown) | ||
|
|
||
| ***Note:*** Unfortunately, this is just a hobby of ours and not a full-time job, so we'll do our best to keep things up to date, but no guarantees. That being said, thanks to everyone for your continued help and feedback as it is really appreciated. We will try to address everything as soon as possible. | ||
| * [Dongdong Wang](https://github.com/dd604) | ||
|
|
||
| ## References | ||
| - Wei Liu, et al. "SSD: Single Shot MultiBox Detector." [ECCV2016]((http://arxiv.org/abs/1512.02325)). | ||
| - [Original Implementation (CAFFE)](https://github.com/weiliu89/caffe/tree/ssd) | ||
| - A huge thank you to [Alex Koltun](https://github.com/alexkoltun) and his team at [Webyclip](webyclip.com) for their help in finishing the data augmentation portion. | ||
| - A list of other great SSD ports that were sources of inspiration (especially the Chainer repo): | ||
| * [Chainer](https://github.com/Hakuyume/chainer-ssd), [Keras](https://github.com/rykov8/ssd_keras), [MXNet](https://github.com/zhreshold/mxnet-ssd), [Tensorflow](https://github.com/balancap/SSD-Tensorflow) | ||
| - Shifeng Zhang, et al. [Single-Shot Refinement Neural Network for Object Detection](https://arxiv.org/abs/1711.06897), [official Caffe code](https://github.com/sfzhang15/RefineDet) | ||
| - [SSD in PyTorch](https://github.com/amdegroot/ssd.pytorch) | ||
| - [Faster RCNN in PyTorch](https://github.com/jwyang/faster-rcnn.pytorch) |
Large diffs are not rendered by default.
Oops, something went wrong.