Dynamic patch in the eval5 single object tracking scenario

[ShengYun-Peng]

After generating the undefended eval5 single object tracking dataset, we’ve observed that the patch’s location and shape seem to shift around across frames compare to the green screen in some videos, such as adversarial video 3 as attached, and we wonder what may be the cause and how we may spatially keep it stationary in the virtual world.

video_adversarial_with_boxes.mp4

[lcadalzo]

I believe this is related to this PR in which @yusong-tan updated the AdversarialTexture attack to support per-frame perspective transform of the patch. As noted in this comment, the Eval 5 dataset differs in that it includes camera motion

@yusong-tan do you know if the updates made in this separate PR (which went into Armory 0.15.1, but not 0.15.0) are relevant here too? It does seem like there are frames for which the patch is not adequately covering the green-screen, including the point at which the patch seems to mostly disappear

[davidslater]

It might be helpful to have the frame-by-frame RGB outputs to rule out any mp4 compression artifacts.

[yusong-tan]

@ShengYun-Peng @lcadalzo The "flickering" of patch in the video is due to imperfect estimation of the four corners of the green screen. In some frames, one or more corners are occluded by pedestrians so their estimates are incorrect and so the patch transforms are affected as a result. In future releases, we'll try to smooth out these estimation errors using some kind of regression, but there likely will always be some residual "flickering" under occlusion. Currently, there is no PR addressing this issue.

[lcadalzo]

After experimenting with eval6/carla_mot/carla_mot_adversarialpatch_undefended.json and eval6/carla_mot/carla_mot_dpatch_undefended.json, I did not notice this to be an issue with the new dataset