[QUESTION] single_texts vs group_texts

[agademic]

Hi @mallorbc!
Thank you so much for your work on the repo and your tutorials!

Did you experiment with the different data preprocessing settings (single_texts, group_texts) in different task scenarios?
I am running experiments on the quotes dataset and I am getting very different losses in these two settings: single ~ 0.89 vs group ~ 3.3.

Single is padded to a certain length, while group is concatenated with eos token in between.

Do you have any idea when to use which setting or why there is this difference in loss?

Any hints are appreciated!

[mallorbc]

While I have not tested the two, intuitively they make sense. For group texts, if your data entries are not related at all, by grouping them together the model will incorrectly learn that some text follows some unrelated text.

If entries are statistically independent, I keep them separate. If they are related, such as a book or some general knowledge corpus for further pretraining, I would group them

[mallorbc]

Closing. Feel free to reopen if you are still confused.