Disable augmentation in training pipeline

[ItsMeTheBee]

Prerequisites

Please answer the following question for yourself before submitting an issue.

• [ x] I checked to make sure that this issue has not been filed already.

1. The entire URL of the documentation with the issue

https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html

2. Describe the issue

Hey there!

Not sure if this is a bug because nobody seems to have an issue with this so I guess I just can´t find the right settings.
I followed the linked tutorial to train the SSD ResNet50 V1 FPN 640x640 model with my own dataset (only 1 object).
After monitoring the training with tensorboard I saw that the images used for training are very distorted.
I deleted all the lines responsible for the augmentation but the training images remain distorted.

Images during training look like this

The images in my dataset look like this

I already asked on stackoverflow but there aren´t many views ans no answers :b
https://stackoverflow.com/questions/63523996/disable-augmentation-in-tensorflow-training-pipeline

So I was wondering how to fix this since I can´t figure out whats going on here.
I already trained a darknet yolo model with this dataset and the results are very good.
But here the map is close to zero and i figured thats because of the distortion in my training images.
If I use some training images for the map calculation the images are displayed correctly with tensorboard.

This issue seems to be related.

[stephanmeyer-tomtom]

Hey,

I face the same issue. Even with no augmentation the images look darker then they should be. I also checked the tf-record files. In them the images are correctly saved.

Any ideas where this may come from?

EDIT: It seems that it's just a visualization issue which doesn't affect training. See #9115 (comment)

[LackesLab]

You should normalize your inputs. After normalization between 0 and 1, it should displayed correctly.

This step is also recommended to improve the learning process. By normalizing inputs, you prevent your training from exploding gradients

[ymodak]

For more context also refer this thread

[ItsMeTheBee]

Do I need to normalize my training dataset or is there a function in the training pipeline i can simply add to achieve this without modifying all of my images?

[LackesLab]

@ItsMeTheBee you can add the following to your training pipeline to rescale all images:

https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing/Rescaling?hl=de

inputs = Rescaling(scale=1.0 / 255)(input)

[ItsMeTheBee]

@CptK1ng Thanks a lot for your answer!

This is based on training using tf.keras right?
Because right now I´m only working with the TensorFlow Object Detection API to train networks and I´m not really familiar with using a python script and writing the pipeline myself (but I would be very glad if you could point me towards a tutorial/ repo for EfficientDet or MobileNet to train object detection that uses regular scripts instead of the Object Detection API - I couldn´t find anything that works anymore or only things that train classification and not detection)

Can I add something like

data_augmentation_options {
    normalize_image {
	original_minval: 0
	original_maxval: 255
	target_minval: 0
	target_maxval: 1
    }
  }

in the pipeline achieve the same thing?

This doesnt work though, I only get black images in tensorboard (which I understand, I just don´t know how to change the parameters to get it right)

[LackesLab]

I have not train a tf2 object detection api model yet, just tf1 versions. Maybe you can try to normalize the image when you create your tf record. Honestly, i have no glue.