Why only last token used in datasets for training?

[VityaVitalich]

Dear maintainers,

Thank you for your awesome paper and open-source project. I recently ran into the detail of dataset pre-processing, which i can not understand properly.

In the processing of datasets you ignore all the labels exept the last one (as it is in

AQLM/src/datautils.py

Line 51 in bf84f39

tar[:, :-1] = -100

). This seems a bit odd as long as transformers perform causal masking inside forward and during training LM model we want to propagate loss through all of the tokens and not the last one.

Could you please explain this detail?

[Godofnothing]

@VityaVitalich actually it is an artifact from GPTQ code. tar is never used in our code (as well as in the original repository).