Does iCAN result contains human-object interactioness relation?

[xiadingZ]

Does iCAN detection result contains human-object relation? then you needn't estimate which object one person interacts with,
Only need to predict which part interacts with this object?

[DirtyHarryLYL]

For part-level interaction, e.g., recognizing hand is interacting with which object and performing what actions, are separated with the instance-level interaction detection.
In the part-level model, we also provide positive and negative training samples for the PaSta recognizer similar to the instance-level model.
For example, given the same human-mug box pair, the instance model may infer that they are non-interactive, but the PaSta model may infer that the hand and head are interacting with the object and thus conclude the result ''drink with mug''. This will bring in more flexibility: if the instance model makes a mistake, the PaSta model may not.

[xiadingZ]

Do you use pretrained instance-level interaction detection? or train instance-level interaction detection first, then part-level? Or simultaneously? In this repo, I only see part-level interaction detection using this:
vec_cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=label_vec, logits=cls_score_vec).
Or can you figure out the code of instance-level interaction detection? thanks very much

[DirtyHarryLYL]

In training, we only train the part model solely. In inference, part model would give the part-level prediction and then late fused with the instance-level prediction from an off-the-shelf instance model like iCAN or TIN.

[DirtyHarryLYL]

For example, the result of TIN is loaded from a pkl here:

HAKE-Action/-Results/Generate_detection.py

Line 28 in e74edac

score_H = pickle.load(open('TIN/score_H.pkl', 'rb'))

[xiadingZ]

In inference, for one human, iterate all objs, late fuse part-obj prediction score and instance-level hoi predication score, if one obj-part score is higher than threshold, then use this human-obj pair to predict action and calculate metric. if all obj-part pairs scores are lower than threshold, then drop this human-obj pair.
Is my understanding correct? and can you give the equations of late fusion

[Foruck]

The equation is : sHO = (((sH + sO) * ssp + sP + sA + sL) * score_I[im_index[keys[obj_index]], x:y]) * hod.
sH, sO and ssp are prediction probabilities from TIN, (sH + sO) * ssp is the instance-level score and (sP + sA + sL) is the part-level score. score_I is the image-level score. hod is the multiplication of human detection score and object detection score (both from the object detector).
In inference, for one human, iterate all objs, and perform the above late-fusion. If the human detection score or object detection score is lower than a given threshold, the pair is dropped. If the interaction score (from TIN, saved in -Results/TIN/neg.pkl and -Results/TIN/pos.pkl) doesn't satisfy the NIS rule, the pair is also dropped.