Maximum number of bounding boxes are predicted with high confidence (99%) when loss_type=LOCALIZATION

[FSet89]

System information

• What is the top-level directory of the model you are using: N/A
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.14.0
• Bazel version (if compiling from source): N/A
• CUDA/cuDNN version: 10 / 7.4.1
• GPU model and memory: Titan V, 12 GB
• Exact command to reproduce: python3 object_detection model_main.py --model_dir MYMODELPATH --pipeline_config_path MYCONFIGPATH

Describe the problem

I am training a MobileNetV2 SSD on my dataset containing only one class. If I set the loss_type in HardNegativeMiner to CLASSIFICATION or BOTH, during validation I see that 0 or 1 bounding boxes are predicted with high confidence. If I set it to LOCALIZATION (which should be more reasonable since I am not interested in classification here), the maximum allowed number of bounding boxes are always predicted, for every image, since the first validation step. At the beginning they are in totally wrong position, then they start to center around the correct object, but they are always the maximum allowed number, even in images without objects. Therefore I have many false positives and, even if some bounding boxes are correct, the others are mostly inaccurate. Why is this happening?

My dataset contains about 10000 images and I am training from scratch.