Bug in Validation Logger

[yashkant]

Hi,

In the line below we aren't multiplying the score with batch-size, and while logging we are dividing the sum of scores with the complete dataset-size. Does this seem correct?

vilbert-multi-task/vilbert/utils.py

Line 271 in a290ba6

self.task_score_val[task_id] += score

vilbert-multi-task/vilbert/utils.py

Line 339 in a290ba6

score = self.task_score_val[task_id] / float(self.task_datasize_val[task_id])

[vedanuj]

This is because the validation method ForwardModelsVal returns a batch score (not divided by the batch size). Here :

vilbert-multi-task/vilbert/task_utils.py

Lines 122 to 163 in a290ba6

	if task_cfg[task_id]["type"] == "VL-classifier":
	loss = task_losses[task_id](vil_prediction, target)
	loss = loss.mean() * target.size(1)
	batch_score = compute_score_with_logits(vil_prediction, target).sum()
	if task_cfg[task_id]["type"] == "VL-classifier-GQA":
	loss = task_losses[task_id](vil_prediction_gqa, target)
	loss = loss.mean() * target.size(1)
	batch_score = compute_score_with_logits(vil_prediction_gqa, target).sum()
	elif task_cfg[task_id]["type"] == "VL-logit":
	vil_logit = vil_logit.view(batch_size, num_options)
	loss = task_losses[task_id](vil_logit, target)
	_, preds = torch.max(vil_logit, 1)
	batch_score = (preds == target).sum()
	elif task_cfg[task_id]["type"] == "V-logit":
	loss = task_losses[task_id](vision_logit, target)
	loss = loss.mean() * target.size(1)
	_, select_idx = torch.max(vision_logit, dim=1)
	select_target = target.squeeze(2).gather(1, select_idx.view(-1, 1))
	batch_score = torch.sum(select_target > 0.5).item()
	elif task_cfg[task_id]["type"] == "V-logit-mc":
	vision_logit = vision_logit[:, 101:]
	vision_logit = vision_logit.squeeze(2).gather(1, multiple_choice_ids)
	vision_logit = vision_logit.unsqueeze(2)
	loss = task_losses[task_id](vision_logit, target)
	loss = loss.mean() * target.size(1)
	_, preds = torch.max(vision_logit, dim=1)
	_, target = torch.max(target, dim=1)
	batch_score = (preds == target).sum()
	elif task_cfg[task_id]["type"] == "VL-binary-classifier":
	loss = task_losses[task_id](vil_binary_prediction, target)
	loss = loss.mean()
	batch_score = compute_score_with_logits(vil_binary_prediction, target).sum()
	elif task_cfg[task_id]["type"] == "VL-tri-classifier":
	loss = task_losses[task_id](vil_tri_prediction, target)
	loss = loss.mean()
	batch_score = compute_score_with_logits(vil_tri_prediction, target).sum()

[yashkant]

Oh, that makes sense! Thanks for explaining.