Problem with ImageRecordIter behavior for image processing (especially when using mean_r, mean_g, mean_b)

[MathieuCiancone]

Hi there,

I'm a bit struggling with the utilization of the .rec files that are fed to mxnet neural_networks. I have encountered weird beahaviors of the mx.io.ImageRecordIter parameters.

The .rec file of my image dataset has been made using the im2rec.py script. First of all, I realized that the script outputs images in a 3,224,224 shape (which is great). Thought, as @anotinelg relized here #15710 (comment) the images are output as BGR or RGB format, depending on the utilisation of the "--pass-through" parameters, which is used to skip all preprocessing steps. From what I understood, when utilizing "--pass-through", images are not processed by opencv and then keep the RGB format. I dunno if this is on puprose or not but it is suprising.

Consequently, I deciding to leave the images untouched and do the preprocessing using ImageRecordIter. Here is the part where i create the iterator, indicating the mean and std values to be used for the normalization of each channel.

valid_iter = mx.io.ImageRecordIter( path_imglist='Input/train_val.lst', path_imgrec="Input/train_val.rec", path_imidx="Input/train_val.idx", data_shape=(3,224,224), scale=1/255, std_r=0.229, std_g=0.224, std_b=0.225, mean_r=0.485, mean_g=0.456, mean_b=0.406, batch_size=64, rand_crop=False, rand_mirror=False, shuffle=False, preprocess_threads=4, prefetch_buffer=1, ctx=str(ctx)[:3], dtype="float32")

When analyzing the result of this preprocessing I was suprised that the result is different than when using :

normalized = mx.image.color_normalize(t.astype(np.float32)/255, mean=mx.nd.array([0.485, 0.456, 0.406]), std=mx.nd.array([0.229, 0.224, 0.225]))

The outputs are :
`[[[3.879014 3.9836137 4.053926 ]
[3.673517 3.7735298 3.844776 ]
[4.3071327 4.4563026 4.4373684 ]
...
[2.3549106 2.4605045 2.607303 ]
[2.3549106 2.4605045 2.607303 ]
[2.3549106 2.4605045 2.607303 ]]

[[4.221509 4.3337536 4.40251 ]
[3.8961387 4.0011206 4.0713553 ]
[1.8069185 1.9002802 1.9798521 ]
...
[2.4405346 2.5480394 2.694449 ]
[2.1665385 2.2679274 2.4155817 ]
[1.9610413 2.0578432 2.2064316 ]]

[[4.2728834 4.421289 4.4373684 ]
[3.2282732 3.3533616 3.426475 ]
[0.8308075 0.90238094 0.9863879 ]
...
[2.1665385 2.2679274 2.3807235 ]
[2.0466652 2.145378 2.2935774 ]
[2.0466652 2.145378 2.2935774 ]]

...

and

[[ 0.24531215 0.4327732 0.8099348 ]
[ 0.24531215 0.4327732 0.8099348 ]
[ 0.24531215 0.4327732 0.8099348 ]
...
[ 2.1975338 2.4285715 2.64 ]
[ 1.5639181 1.7457983 2.0474076 ]
[ 1.7694151 1.9558823 2.2565577 ]]

[[-0.14855729 0.03011205 0.40906325]
[ 0.05693974 0.2401962 0.6182137 ]
[ 0.33093593 0.5203082 0.8970808 ]
...
[-0.30268008 -0.12745096 0.18248373]
[ 1.7865399 1.9733893 2.2739873 ]
[ 2.11191 2.3060222 2.6051416 ]]

[[-0.06293353 0.11764706 0.49620935]
[-0.06293353 0.11764706 0.49620935]
[ 0.05693974 0.2401962 0.5833553 ]
...
[-1.2787911 -1.12535 -0.8109804 ]
[ 1.1186745 1.3256303 1.6291069 ]
[ 2.1632845 2.3935575 2.64 ]]

...`

After further analysis, I realized that the "mean" parameters of the RecordIter are not taken into account. Thus pixel values are divided by 255 and by each std but the mean of each channel is not subtracted. When looking at the source code, there is indeed no subraction taking into account the 'mean-r' parameters (nor the others).

As both im2rec.py and ImageRecordIter create problems with data processing, i'll just preprocess the data before creating the .rec file.

Though I wanted to report the problem to know if anyone as a solution to this, and for the sake of pointing it out :)