Why use padding_mode='border'?

[kwea123]

When warping the image, you use padding_mode='border', which causes border effects (e.g. on the left and bottom for image 2->1)

monodepth2/trainer.py

Lines 381 to 384 in ec10cf1

	outputs[("color", frame_id, scale)] = F.grid_sample(
	inputs[("color", frame_id, source_scale)],
	outputs[("sample", frame_id, scale)],
	padding_mode="border")

In the code it seems that you compute the reconstruction loss on the whole image, including the border effect parts (if I'm wrong, correct me). Wouldn't it degrades the model's performance?

What I've seen others doing (e.g. struct2depth) is to use a mask, and only compute the loss on the masked (valid) area. This can be done using padding_mode='zeros' to generate the valid mask. E.g., black areas are not contributing to the loss.

What's your opinion?

[mdfirman]

Thanks for the question! I'm going to leave this one for @mrharicot. Thanks!

[mrharicot]

Hi,
Without the border padding the network tends to predict zero disparities on the edge of the image in the stereo case. Similar to this figure, taken from monodepth 1.
It also helped stabilize training in the first few iterations.

We actually haven't evaluated whether it hurts or help depth prediction accuracy.

[kwea123]

Do you have any explanation why it predicts zero disparities on the edge or why it helps "stabilize training"?
For the zero disparities, the loss is computed on both 0->1 and 0->-1 so I think padding zeros on borders can still let the network learn from image 0->1, where no zero or border padding is required?

[mdfirman]

Hi @kwea123 , I don't know exactly why these effects occur, but could take some guesses. It isn't too hard to change the network to have zero padding instead of 'border' padding, so it could be worth investigating this effect yourself. Do report back if you make any interesting findings!

Thanks,

[aj96]

@mrharicot @mdfirman When you used the zero padding as opposed to border padding, were you masking out the corresponding portion for the target frame? If you used zero padding when creating the warped frame, but you didn't mask out the corresponding portion for the target frame, this could lead to strange results. Ideally, you'd just apply the fly-out mask once to the view-synthesis loss, no?

[ClementPinard]

Hello,

be careful with zero padding, due to how interpolation is done, the pixels that are sampled right on the frontier will have a mix of gray (more specifically the color [0,0,0]) and the valid color. you need to explicitely filter out the out of bound pixels and not only padd with zeros. Here is a quick discussion about it : https://discuss.pytorch.org/t/spatial-transformer-networks-boundary-grid-interpolation-behaviour/8891

[mrharicot]

As @ClementPinard mentioned, using padding_mode='zeros' causes a problem at the image edges as the network will essentially learn to do patch comparison and the only location which contains pure black edges is on the edge of the image which causes the network to match the edges of the images together resulting in zero disparity.