SFT of LLama 2

[gsaivinay]

Hello,

Just today, Meta open sourced Llama 2 models. Wondering if OA team is considering these.

[andreaskoepf]

Absolutely. We are already evaluating the models. We will start own fine-tuning runs soon (in a couple of hours).

[Billyroot]

That's good to know. Le mer. 19 juil. 2023 à 00:49, Andreas Köpf ***@***.***> a écrit :

…

Absolutely. We are already evaluating the models. We will start own fine-tuning runs soon (in a couple of hours). — Reply to this email directly, view it on GitHub <#3582 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACVZJTUM2EH3LVCVXGH7TO3XQ4HH5ANCNFSM6AAAAAA2O4RWHI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[gsaivinay]

We will start own fine-tuning runs soon (in a couple of hours).

This is really great. Please keep us posted if possible, on the training and evaluation progress.

[flozi00]

Is there any place to track the training ?
I running some experiments using lora, no resized embeddings for OA special tokens, sequence length of 4096 tokens
The loss of 13b model is stable and lower than 2048 tokens falcon 7b

[gsaivinay]

Is there any place to track the training ?

I'm assuming the training progress will show up here as soon as the process is started, but I might be wrong.

[andreaskoepf]

I'm assuming the training progress will show up here as soon as the process is started, but I might be wrong.

We publish/copy successful runs afterwards to the public-sft wandb project.

[Billyroot]

So we just have to wait for the release :). Do you think it might happen next week?

[gsaivinay]

well, 13b model is here https://huggingface.co/OpenAssistant/llama2-13b-orca-8k-3319 and it is with 8k context length 🤯

[olliestanley]

Here is the first oasst dataset SFT model for Llama2 70B. As usual details are on the HF page and future models will also be on HF, so I will close this issue now

https://huggingface.co/OpenAssistant/llama2-70b-oasst-sft-v10