MVM for CLIP feature

[vateye]

Hi, I would like to know how to compute the loss between VideoSwin and the CLIP features in latest paper. Since the Swin family models take the patch size as 4x4, however for ViT the patch size is 16. I would like to know how to compute the loss between these two? (i.e., l1 loss)?

Thanks.

[tsujuifu]

Hi vateye,

Thanks for your interest in our empirical study.

We use VidSwin and CLIP (ViT-B/32), both with a patch size of 32.
For example, a 224² video frame will be 7² visual features.

[vateye]

Thanks! Meanwhile, I am curious how do you apply Random Masking and Blockwise Masking together with the ratio of 30%. Is there any code for doing this? Since these two are disjoint masking strategies.

[tsujuifu]

This code only has BM and AM.
During pre-training, we randomly chose among them. You can also use a different probability.