customized inputs

[Mingyuan1997]

Is there any suggestion on how to run the model on customized input videos?

Thank you!

[yrcong]

Hi,
i think it is not difficult to use the model to infer the customized videos.
Line 167-178 in https://github.com/yrcong/STTran/blob/main/dataloader/action_genome.py may help you:)
best

[xiaodanhu]

If we test on custom video, it seems that the information like attention_relationship and bboxes are required?

[Mingyuan1997]

I think so.

[yrcong]

If we test on custom video, it seems that the information like attention_relationship and bboxes are required?

Why? The attention relationships should be predicted and the bboxes should be inferred by the object detector.

[xiaodanhu]

If we test on custom video, it seems that the information like attention_relationship and bboxes are required?

Why? The attention relationships should be predicted and the bboxes should be inferred by the object detector.

Thanks for replying!
When I checked the dataloader in action_genome.py, the person_bbox and object_bbox were loaded from person_bbox.pkl and object_bbox_and_relationship.pkl, respectively. So I was guessing, if loading the customized video, we also need this information so that we can load it properly. Can you indicate where the model automatically predicts the relationship and bboxes? Thank you very much!

[yrcong]

If you just want to test on your customized video dataset, only the video frames (self.video_list in the class AG) are necessary (for the setting SGDET).
Sometimes person_bbox and object_bbox are used in the test because there are other two settings PredCLS/SGCLS which are widely used in the image scene graph generation.