Concerning parsing prediction

[Daniellli]

Hi, thank you for your wonderful work.

I am trying to parse the prediction, and i feel confusing at the following lines.

butd_detr/train_dist_mod.py

Lines 227 to 232 in 10570e0

	end_points['last_sem_cls_scores'] = sem_scores
	# end contrast
	sem_cls = torch.zeros_like(end_points['last_sem_cls_scores'])[..., :19]
	for w, t in zip(wordidx, tokenidx):
	sem_cls[..., w] += end_points['last_sem_cls_scores'][..., t]
	end_points['last_sem_cls_scores'] = sem_cls

Why replacing the last_sem_cls_scores to the new generated tokenidx and wordidx based class info? what do the tokenidx and wordidx mean?

thank you for your attention

[Daniellli]

moreover, does this line right?

dividing the probability of being object, the top class token with highest response will be changed, maybe become the second largest one?
https://github.com/nickgkan/butd_detr/blob/10570e0b6826d4a236b18c2c8fac5903866e1c60/models/ap_helper.py#L150C20-L150C20

[ayushjain1144]

Why replacing the last_sem_cls_scores to the new generated tokenidx and wordidx based class info? : The idea is to convert the logits over the detection prompt span (i.e. 256 dimensions) to logits over class labels (18 classes and thus 19 dimensions to include no object class).

what do the tokenidx and wordidx mean?: See #17

moreover, does this line right?: This line ends up being inconsequential because here we multiply back sem_cls_scores with obj_prob. We will fix this in the next version.

[Daniellli]

Hi,

does the line 153 correct? if it is correct, the boxes with high IoU but diffferent prediction class will be maintained, which i think a little unreasonable if the IoU is extremely high, such as 1?

[Daniellli]

moreover, the line 155 make me confusing, why the [last - 1] are always excluded? is there any probability about the [last -1] box is the correctest one?

[ayushjain1144]

nms.py comes unchanged from votenet, I believe they would be able to better assist you. We didn't really look deep into the nms code.

[Daniellli]

hi , i am little confusing with following lines:

butd_detr/src/grounding_evaluator.py

Lines 197 to 201 in 10570e0

	pmap = positive_map[bid, :num_obj]
	scores = (
	sem_scores[bid].unsqueeze(0) # (1, Q, 256)
	* pmap.unsqueeze(1) # (obj, 1, 256)
	).sum(-1) # (obj, Q)

why the GT involved in parsing prediciton ?