why keep "###" before instruct text?

[Cescfangs]

Video-LLaMA/video_llama/datasets/datasets/video_instruct_dataset.py

Lines 250 to 252 in 1728e14

	for tokenized_len, speaker in zip(tokenized_lens, speakers):
	if speaker == "human":
	target[cur_idx+2:cur_idx + tokenized_len] = IGNORE_INDEX

I was reading _mask_targets(), I guess this function is using mask to ignore loss from instruct text, but why do you manually keep [curr_idx: curr_idx+2] which is "###" before the actual instruct text?

[hangzhang-nlp]

The assistant will learn to generate "###" if it wants to end the current round. So "###" can be understood as the EOS flag of each session.

[Cescfangs]

The assistant will learn to generate "###" if it wants to end the current round. So "###" can be understood as the EOS flag of each session.

Thanks for the reply, I understand "###" works like EOS here, and I think the "###" in assistant text is more like EOS? howerver why do we want assistant to learn EOS before instruct?

[hangzhang-nlp]

During inference, we will stop generating tokens once the assistant outputs "###".
Suppose that the training data is " ###Human: Hi. ###Assistant: Hi, can I help you. ###Human: yes".
The content which needs to calculate loss is "Hi, can I help you. ###". So the model can learn to generate "###" as the end flag of his reply.

[Cescfangs]

During inference, we will stop generating tokens once the assistant outputs "###". Suppose that the training data is " ###Human: Hi. ###Assistant: Hi, can I help you. ###Human: yes". The content which needs to calculate loss is "Hi, can I help you. ###". So the model can learn to generate "###" as the end flag of his reply.

I mean you are also keep the first "###"(the one before "Human: Hi...").

[hangzhang-nlp]

Oh, I see. This place is just for convenience, and you can add a judgment, mask the first "###".

[Cescfangs]

Okay, thanks for the confirmation