[tokenization] preprocessing inputs and labels

[ArashAhmadian]

Hello,

Firstly thanks for open-sourcing all components of alpaca_farm!

I'm looking into data_preprocessor.py and am wondering where/if the labels are set to the input_ids shifted by 1. (something like input_ids = input_ids[...,1:] labels = input_ids[...,:-1]classic next token prediction)

However, it seems like they're set to input_ids without any shifting? I'm not sure what I'm missing but any clarification would be great :)

[ArashAhmadian]

NVM, for baseline sft, since a default hugging-face trainer is used, shifting happens inside the default forward implementation :))

[lxuechen]

Yeah, you’re right. Shifting is handled internally in the trainer.