This is an attempt to replicate the results of a very interesting idea proposed by Zhao Chen, the main concept is to combine different tasks into a single topology, those tasks are edge or contour extraction, object segmentation and object detection. All these tasks can collaborate with each other in order to have a higher accuracy when detecting and segmenting objects in a scene. The basic concept is presented in the following figures:
The reasoning behind is simple, the features extracted by some of the tasks where smaller receptive fields are required can be used to enhance those tasks where the receptive fields required is large. Therefore the model key idea is summarized as follows:
-
Use a basic classification model to extract features that could be shared across different tasks.
-
The edge detection task should enhance the prediction of the segmented object in the scene
-
The segmentation task should enhance the prediction of the Bounding boxes extracted by the model.
-
Bounding boxes extracted should in theory be much better than performing the tasks alone, since segmentation and edge detection have shared their information.
-
For backpropagation the errors from Bounding boxes -> segmentation -> edge-detection shall provide a better learning strategy for the model.
I have been working on this for a couple of months now, the code is far from finished, let alone optimized, any collaboration or comment would be appreciated. I have been using Tensorflow, and coco API for data handling and model training.
The creator of the model tested it on cityscapes dataset, my purpose is not only to replicate the results, but also to use COCO dataset with this model. So far I have implemented three different models for feature extraction shared across tasks or body network: Resnet, SqueezeNet, Helnet. As for the Bounding box detector a gridbox adapted from detectnet_coverage_rectangular.cpp was implemented here, very similar to what DetectNet has.
Just type "python main.py --help" to see the different options, it is very important to notice that this code is under heavy development. Therefore you may need to review the code to understand how to run it successfully.
-
Datasets: training/testing/validation datasets have to be places inside "datasets/" subfolder and the "config.py" file modified accordingly
-
Models and Logs: model checkpoints and tensorboard logs have to be places inside "models/" and "tfboard" respectively.
-
Debuging: two folders are provided for you to analyze your dataset and information extracted such as masks and Bboxes, the folder are "debug_images" and "debug_masks".
-
Training status: So far I have been able to train the model on both COCO and CityScapes however my results are far from those shown in here, consequently I am spending some time now to debug the code and to see whether I can improve its performance.
Josue R. Cuevas