Fixed [CLS] token during inference?

[dwang-sflscientific]

Hi

For inference, the CLS token(L157 and L160 in train.py) is still basing on ground-truth label, should they be static CLS token instead?

[dwang-sflscientific]

For pretraining&fine-tuning, don't understand why ground truth labels are used as [CLS] token as well.

[somepago]

Hi, you don't need to base the CLS token on the actual label. It's an artifact from another project, the static token is generated in the embed_data_mask function (L321).

[dwang-sflscientific]

Ah I see. Thanks for the explaianation.