label_map does not do the same augmentation (random crop) as the input image

[haooooooqi]

Hi
Thanks so much for the nice work!
I am curious if you could share the insight on processing of the label_map.
If I understand it correctly, after we load image and the corresponding, we shall do the same cropping/ flip/ resize, but in

TokenLabeling/tlt/data/label_transforms_factory.py

Lines 58 to 73 in aa438ef

	def __call__(self, img, label_map):
	i, j, h, w = self.get_params(img, self.scale, self.ratio)
	coords = (i / img.size[1],
	j / img.size[0],
	h / img.size[1],
	w / img.size[0])
	coords_map = torch.zeros_like(label_map[0:1])
	# trick to store coords_map is label_map
	coords_map[0,0,0,0],coords_map[0,0,0,1],coords_map[0,0,0,2],coords_map[0,0,0,3] = coords
	label_map = torch.cat([label_map, coords_map])
	if isinstance(self.interpolation, (tuple, list)):
	interpolation = random.choice(self.interpolation)
	else:
	interpolation = self.interpolation
	return torchvision_F.resized_crop(img, i, j, h, w, self.size,
	interpolation), label_map

Seems only image was cropped, but the label map does not do the same cropping, which make the label map not match with the image?

Shall we do

        return torchvision_F.resized_crop(
                img, i, j, h, w, self.size, interpolation
        ), torchvision_F.resized_crop(
                label_map, i / ratio, j / ratio, h / ratio, w / ratio, self.size, interpolation
        )

Thanks

[zihangJiang]

Thanks for your question, the coords (i.e. i,j,h,w) for the random crop are stored in the label map and will be used later here.

TokenLabeling/tlt/data/mixup.py

Lines 48 to 70 in aa438ef

	def get_labelmaps_with_coords(label_maps_topk, num_classes, on_value=1., off_value=0.,label_size=1, device='cuda'):
	'''
	Adapted from https://github.com/naver-ai/relabel_imagenet/blob/main/utils/relabel_functions.py
	Generate the target label map for training from the given bbox and raw label map
	'''
	# trick to get coords_map from label_map
	random_crop_coords = label_maps_topk[:,2,0,0,:4].view(-1, 4)
	random_crop_coords[:, 2:] += random_crop_coords[:, :2]
	random_crop_coords = random_crop_coords.to(device)
	# trick to get ground truth from label_map
	ground_truth = label_maps_topk[:,2,0,0,5].view(-1).to(dtype=torch.int64)
	ground_truth = one_hot(ground_truth, num_classes, on_value=on_value, off_value=off_value, device=device)
	# get full label maps from raw topk labels
	label_maps = get_featuremaps(label_maps_topk=label_maps_topk,
	num_classes=num_classes,device=device)
	# get token-level label and ground truth
	token_label = get_label(label_maps=label_maps,
	batch_coords=random_crop_coords,
	label_size=label_size,
	device=device)

This helps to crop the label map using the given coords in parallel with rio_align function, which will be slightly faster than processing each label map individually as in your example.