Question about merlot model

[ZihaoZheng98]

Dear Rowan,
Hi, I have noticed this paper recently, I really think this paper is of great value, I understand nearly all the details of your paper except the model. I know the details are in the code, but I am not familiar with TensorFlow, if you can explain these to me, I will understand the code much easier, so I wonder if you can answer my questions when you have time?

1.What does chunk mean in the code? Does it represent the max number of segments a video has been segmented?
2.In 3.2, you said that MERLOT takes multiple unordered video frames as input, but in Joint Vision-Language Encoder
part, you say that position embeddings are added to the vision components, do you mean that, when fed into the model, the image and the corresponding sentence have the same position embedding?
3.In 3.3, Temporal Reordering part, I understand the core idea, but I am not sure about your methods, is it correct that you randomly choose i frames, and then change the position embedding of these frames to the same embedding [image_unk_0]?

Best regards,
Zihao

[rowanz]

thanks for all the kind words!

1. Chunk in the code means segment in the paper, yeah :)
2. We experimented with a few different ways but what we found works best (and is in the code/paper) is where the sentence gets the normal position embedding, and the image gets a permuted position embedding (ie [image_unk_0]).
3. Kinda -- so we do choose i frames, however we give them different position embeddings so the model can tell them apart, ie, if there are 4 frames in total and we want to permute 2 of them we might give them position embeddings

[pos0], [image_unk_1], [image_unk_0], [pos3]

(Written in the right order).
and the model would have to learn in this case, for instance, that pos0 < image_unk_1 < image_unk_0 < pos3.

hope that helps :)

[Haelles]

Dear Rowan,
Thanks for your great work! However, I still have some questions, if you can answer my questions when you have time?

1. I am very confused with [image_unk_1], [image_unk_0]. The paper says "by replacing the segment-level position embeddings (e.g. [image_t]) for that frame with a random and unique position embedding, e.g. [image_unk_0])." I think that sometimes image_unk_1 < image_unk_0, and some other time image_unk_0 < image_unk_1?
2. Where is the code about scrambling video frames and adding these embeddings? I find

   merlot/model/modeling.py

   Line 321 in fc77f99

   my_pe = one_hot_gather(img_pe_table, tf.reshape(shuffled_idx_img, [-1]))

   but I dont't know why this function can generate embeddings like [image_unk_0].

Thanks a lot!!

[rowanz]

1. Yep exactly, those are just IDs and their ordering can vary.

2. sure! shuffling happens here.

[Haelles]

1. Yep exactly, those are just IDs and their ordering can vary.
2. sure! shuffling happens here.

Thank you!!