Training recipe??

[milsun]

The blog says training recipe is too released in the code, but I cannot find it. Can you update the repo with code used for training the model, along with required dependencies/guide, etc, to help us do the same, maybe with bigger models.
Thanks for this awesome repo.

[lxuechen]

Hi, thanks for your interest!

We will release the training code once the Hugging Face interface to LLaMA becomes stable (merged into main).

Our fine-tuning procedure is standard and was performed with huggingface's trainer. You can see our hyperparameters here.

[Hiusam]

Hi, do you fine-tune the model using Next Token Prediction as the pre-training? And how much CUDA memory does the training cost? Is it possible to do the training using only one 80G A100 GPU?

[rtaori]

Hi all,

We have released the training code, see https://github.com/tatsu-lab/stanford_alpaca#fine-tuning. Please open a new issue for any further questions/concerns.

[pGit1]

@rtaori as far as I can tell from your code it looks like standard teacher forcing (aka next token prediction), is this accurate?