Support fine-tuning LLaMA3?

[cnlinxi]

Very great project!

I tried to use training/bash/run_ds3.sh fine-tuning LLaMA3-8b. However, I found that the following code does not work for LLaMA-3 during the debugging process. Is this in line with expectations?

LLMBox/training/dataset/sft_dataset/sftdataset.py

Line 64 in 1f4fe29

tokenizer.add_eos_token = True

Thanks for your reply.

[timturing]

Thank you for your issue! I have tested the model, and it appears to be working well.

Here's the input_id of the first example from the Alpaca dataset:

The LLaMA3-8b tokenizer configuration is as follows:

128000: AddedToken("<|begin_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
128001: AddedToken("<|end_of_text|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),

This indicates that the eos_token has been added successfully.

The training loss is also within the normal range:

{'loss': 1.5548, 'grad_norm': 18.375739200079522, 'learning_rate': 1e-05, 'epoch': 0.0}    0%|                                                     | 1/1200 [00:13<4:20:01, 13.01s/it]
{'loss': 1.3182, 'grad_norm': 10.779239457063062, 'learning_rate': 1e-05, 'epoch': 0.0}    0%|                                                     | 2/1200 [00:20<3:19:55, 10.01s/it]

There is one possible reason it might not be working for you: the tokenizer saves the tokenized data file into a .pth file directly and reloads it. We set this to save time when re-training large datasets. However, if you load a completely new model or a new tokenizer, it may not be compatible with the previously tokenized data (e.g., the eos_token might not match in this case).

If this is your situation, try deleting the .pth data file and retrain the model. We are working on fixing this issue soon.

Please let me know if you continue to experience the problem.

[cnlinxi]

@timturing
Thanks for your reply.
In fact, I found a similar issue: huggingface/transformers#30947

the tokenizer for Llama3 is a PreTrainedTokenizerFast, not the LLamaTokenizer or a LlamaTokenizerFast. Though it might actually be good to support an easy way to add bos and eos. Currently what you have to do is update the TemplateProcessor which is fairly annoying (not beginner friendly).
huggingface/transformers#30947 (comment)

LLaMA3-8b has been fine-tuned according to the solution in this issue (Otherwise, the fine-tuned model cannot be stopped normally), but there are still some strange prefixes when inference, which may still be related to tokenizer.

After all, according to the method in this issue, eos token will be always added.

That is,

tokenizer.add_eos_token = True 

or

tokenizer.add_eos_token = False

doesn't work.

How should this be solved? Thank you.

[timturing]

Thank you for providing the additional information. I have observed the issue as well. As you mentioned, this cannot be resolved with our current code. Should the transformers team develop a solution, we will promptly update our code to incorporate it.

[cnlinxi]

Thank you for providing the additional information. I have observed the issue as well. As you mentioned, this cannot be resolved with our current code. Should the transformers team develop a solution, we will promptly update our code to incorporate it.

Ok. Closing this issue. Looking forward to fix.