Number of AvgNumGroundtruthBoxesPerImage is always 100

[christian-lanius]

System information

• What is the top-level directory of the model you are using: object_detection/models/research/object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Kubuntu 18.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.14
• Bazel version (if compiling from source): N/A
• CUDA/cuDNN version: Cuda 10.0 cuDnn7
• GPU model and memory: RTX 2080TI
• Exact command to reproduce:

PIPELINE_CONFIG_PATH=/home/lanius/code/object_detection/trained_models/ssd_resnet50_v1_fpn_shared_box_predictor/pipeline_clean_few_classes.config
MODEL_DIR=/home/lanius/code/object_detection/results/ssd_resnet50_v1_fpn/clean_few_classes
SAMPLE_1_OF_N_EVAL_EXAMPLES=1
python object_detection/model_main.py \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --model_dir=${MODEL_DIR} \
    --sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
    --alsologtostderr 

Describe the problem

I am attempting to train the ssd_resnet_50_fpn_coco object detector on a simplified coco dataset, however the same issue persists even if I use the all coco classes. The metrics reported by tensorboard look off:

The number of AvgNumGroundtruthBoxesPerImage is always 100, the maximum number of output boxes from the model.
Indeed in the tensorboard display, there are some images with no GT bboxes:

While the loss in general decreases, the performance of the detector is very poor. After 15000 steps with a batchsize of 8:

 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.015
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.031
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.015
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.004
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.031
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.018
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.025
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.026
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.006
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.059

[zhangqianYY]

i meet same priblem ,do you have solve it?

[christian-lanius]

Hi Zhangqian,
After tuning some hyperparameters, I got better performance (not super well, but enough for my needs). I think the problem is mainly due a reporting problem, and not an actual performance problem.
I think at this point, with YOLOv3 impressive performance, there is little reason to use a SSD architecture. YMMV

[jveitchmichaelis]

This is due to a parameter in the pipeline.config called unpad_groundtruth_tensors which I assume pads this list to whatever the maximum number of boxes is. Basically the stats in TargetAssignment give you information about how ground truth bounding boxes are being assigned to anchors in the model. These numbers don't affect training in any way, but may be useful to catch errors in your dataset and also for model debugging.

If you set this from false to true, you'll get the "right" number being reported. It's simply tracking how many ground truth boxes on average you have in your dataset.

I'm not sure what the purpose of padding this is - there are references to this in the codebase, e.g. here. As it doesn't adversely affect training, I'd leave it on just in case.