Does re-scaling damage the unknown scene coordinate masks?

[qiyan98]

Hi,

Thanks for the wonderful open-sourced project (again)!

I have a question on the potentially harmful effects of label re-scaling. The re-scaling of the image is generally fine. But re-scaling for 3D labels may change the 0 value for invalid scene coordinate masks.

dsacstar/dataset.py

Lines 187 to 199 in 3ffbcb1

	if self.init:
	if self.sparse:
	#rotate and scale initalization targets
	coords_w = math.ceil(image.size(2) / Network.OUTPUT_SUBSAMPLE)
	coords_h = math.ceil(image.size(1) / Network.OUTPUT_SUBSAMPLE)
	coords = F.interpolate(coords.unsqueeze(0), size=(coords_h, coords_w))[0]
	coords = my_rot(coords, angle, 0)
	else:
	#rotate and scale depth maps
	depth = resize(depth, image.shape[1:], order=0)
	depth = rotate(depth, angle, order=0, mode='constant')

In the loss function, the mask is used as follows:

dsacstar/train_init.py

Lines 191 to 192 in 3ffbcb1

	# check for invalid/unknown ground truth scene coordinates (all zeros)
	gt_coords_mask = torch.abs(gt_coords[0:3]).sum(0) == 0

We are concerned about this in our project as the training labels might become accurate after augmentation. I wondered if you have some insights on this issue.

Many thanks!

[qiyan98]

As indicated in the F.interpolate document, the default interpolation mode is nearest. Do you think this could help to justify the regression label re-scaling?

mode (str) – algorithm used for upsampling: 'nearest' | 'linear' | 'bilinear' | 'bicubic' | 'trilinear' | 'area'. Default: 'nearest'

Thanks.

[ebrach]

Hi,

yes, the nearest interpolation is important to not mix zeros (ie invalid labels) and non-zero entries. Results might differ for your specific project, but we have seen no problem with label re-scaling like this. Note, that depending on how you train (RGB mode, or end-to-end training) these labels are only used as a coarse target or an initialisation. The training will refine these labels in most circumstances. The only case where this not happens would we pre-training in RGB-D mode and then omitting end-to-end training.

Best,
Eric