Mean, std normalization inconsistent

[aabbas90]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• [y] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• [y] I am reporting the issue to the correct repository. (Model Garden official or research directory)
• [y] I checked to make sure that this issue has not been filed already.

1. The entire URL of the file to debug:

https://github.com/tensorflow/models/blob/master/research/object_detection/model_lib_v2.py

2. Describe the bug:

The mean and std normalization of COCO images after preprocessing is still not in normal range in CenterNet. On efficientdet_d0_coco17_tpu-32 behaviour is normal and preprocessing is giving close to 0 mean and 1 std.
I have tried two ways to experiment with CenterNet by using centernet_resnet_v1_50_fpn_512x512_coco17 config from the model zoo (without any checkpoint):

1. By mentioning means and std in config:
Means and stds taken as is from pre-trained model config file in the model zoo:

channel_means: [104.01362025, 114.03422265, 119.9165958 ]
channel_stds: [73.6027665 , 69.89082075, 70.9150767 ]

The mean and std of images after normalization are computed from:

models/research/object_detection/model_lib_v2.py

Line 598 in 3f6fe2a

features, labels = data_iterator.next()

features['image'].numpy().mean()
-115.607056
features['image'].numpy().std()
8.1939125

Notice the very negative value of mean and large value of std (batch size: 64).

2. By mean:0, std:1 or not mentioning mean, std in config:

channel_means: [0, 0, 0]
channel_stds: [1, 1, 1]

features['image'].numpy().mean()
-54.237076
features['image'].numpy().std()
80.218575

3. If using efficientdet_d0_coco17_tpu-32 not containing any mean, std info:

features['image'].numpy().mean()
-0.9214938
features['image'].numpy().std()
1.362439

3. Steps to reproduce

Run https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py on config file of centernet_resnet_v1_50_fpn_512x512_coco17 from TF model zoo and coco dataset.

4. Expected behavior

Mean and std normalization should be consistent and inner workings documented. After standardization mean and std should be 0 and 1 respectively.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 18.04): 18.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 2.3.0
• Python version: 3.7

[aabbas90]

@vighneshbirodkar Can you please comment on CenterNet behaviour of preprocessing? Thanks

[vighneshbirodkar]

@aabbas90 Can you share your config that you used to get those means/stds ? Are you trying to use the means/stds used in hourglasss with a ResNet backbone ? That is not the right thing to do, since the ResNet feature extractor does normalizing on its own
https://github.com/tensorflow/models/blob/master/research/object_detection/models/center_net_resnet_feature_extractor.py#L99

If there are some more clarifications needed, feel free to re-open.

[aabbas90]

Hi @vighneshbirodkar ,

Attached is the exact config file I am using. After removing the mean and std values from the config, the mean and std values of the pre-processed COCO data I get is:

features['image'].numpy().mean()
-2.9492028
features['image'].numpy().std()
84.565216

The value of mean looks ok, but I am not sure why is the std not close to zero. Looking deeper, the normalization is happening at:
tensorflow/python/keras/applications/imagenet_utils.py

def _preprocess_symbolic_input(x, data_format, mode):
  """Preprocesses a tensor encoding a batch of images.

  Arguments:
    x: Input tensor, 3D or 4D.
    data_format: Data format of the image tensor.
    mode: One of "caffe", "tf" or "torch".
      - caffe: will convert the images from RGB to BGR,
          then will zero-center each color channel with
          respect to the ImageNet dataset,
          without scaling.
      - tf: will scale pixels between -1 and 1,
          sample-wise.
      - torch: will scale pixels between 0 and 1 and then
          will normalize each channel with respect to the
          ImageNet dataset.

  Returns:
      Preprocessed tensor.
  """
  if mode == 'tf':
    x /= 127.5
    x -= 1.
    return x
  elif mode == 'torch':
    x /= 255.
    mean = [0.485, 0.456, 0.406]
    std = [0.229, 0.224, 0.225]
  else:
    if data_format == 'channels_first':
      # 'RGB'->'BGR'
      if backend.ndim(x) == 3:
        x = x[::-1, ...]
      else:
        x = x[:, ::-1, ...]
    else:
      # 'RGB'->'BGR'
      x = x[..., ::-1]
    mean = [103.939, 116.779, 123.68]
    std = None #<------

  mean_tensor = backend.constant(-np.array(mean))

  # Zero-center by mean pixel
  if backend.dtype(x) != backend.dtype(mean_tensor):
    x = backend.bias_add(
        x, backend.cast(mean_tensor, backend.dtype(x)), data_format=data_format)
  else:
    x = backend.bias_add(x, mean_tensor, data_format)
  if std is not None:
    x /= std
  return x

Notice the part '<------', which is the code getting executed. So the data is not normalized by std, do not know if that is the correct functionality or not.

centernet_resnet_v1_50_fpn_512x512_coco17.zip

[vighneshbirodkar]

You are seeing that because the resnet preprocessed inputs using caffe mode
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/applications/resnet.py#L524
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/applications/imagenet_utils.py#L73

This specifically does not do any rescaling, that's why you observe large values. With that said. the ResNet v1 network which follows later is trained using the same setting and should not have any problems with that input.

[aviraljain99]

Hi @vighneshbirodkar

I am trying to use SSD MobileNet V2 FPNLite 640x640 from the TF2 Object Detection Model zoo and convert it to tflite. I have trained this model on a custom dataset and I am trying to use post-training quantization.

Could you guide me as to how I can find out the correct way to standardize the input to this model which I am specifically interested in? Where can I find what pre-processing is done by the model (or the training process) on the input images? I am also looking into using SSD ResNet50 V1 FPN 640x640 (RetinaNet50) for my task and I am interested in knowing how the images for this model are pre-processed during training. This will be immensely helpful since at present I am getting poor predictions from my model and I believe I am passing inputs to the model differently to how the model was trained.

Thanks,