prompt format?

[silvacarl2]

this is not an issue but did not know where to put it. Is there a specific prompt format to use?

[mallorbc]

Carl,

This is just a base model(for now), so there is no fancy prompting except for BOS and EOS token, which is <|startoftext|> and <|endoftext|> respectfully.

[silvacarl2]

LOL THX

[loofahcus]

Carl,

This is just a base model(for now), so there is no fancy prompting except for BOS and EOS token, which is <|startoftext|> and <|endoftext|> respectfully.

Actually, just EOS (<|endoftext|>) in the base models :)

[mallorbc]

Carl,
This is just a base model(for now), so there is no fancy prompting except for BOS and EOS token, which is <|startoftext|> and <|endoftext|> respectfully.

Actually, just EOS (<|endoftext|>) in the base models :)

The tokenizer has a unique BOS token. It should not used for finetuning?

[loofahcus]

Carl,
This is just a base model(for now), so there is no fancy prompting except for BOS and EOS token, which is <|startoftext|> and <|endoftext|> respectfully.

Actually, just EOS (<|endoftext|>) in the base models :)

The tokenizer has a unique BOS token. It should not used for finetuning?

I think it's OK using it in finetuning.

[ericzhou571]

@loofahcus I carefully check the tokenizer token map and find three speical token <|System|> <|Human|> <|Assistant|>.
Will that be the speical token you plan to use to build multi round conversation?
e.g.:
<|Human|>repeat "this is a multi-turn conversation1" pls<|Assistant|>this is a multi-turn conversation1<|endoftext|><|Human|>repeat "this is a multi-turn conversation2" pls<|Assistant|>this is a multi-turn conversation2<|endoftext|><|Human|>repeat "this is a multi-turn conversation3" pls<|Assistant|>this is a multi-turn conversation3<|endoftext|>

[loofahcus]

@loofahcus I carefully check the tokenizer token map and find three speical token <|System|> <|Human|> <|Assistant|>. Will that be the speical token you plan to use to build multi round conversation? e.g.: <|Human|>repeat "this is a multi-turn conversation1" pls<|Assistant|>this is a multi-turn conversation1<|endoftext|><|Human|>repeat "this is a multi-turn conversation2" pls<|Assistant|>this is a multi-turn conversation2<|endoftext|><|Human|>repeat "this is a multi-turn conversation3" pls<|Assistant|>this is a multi-turn conversation3<|endoftext|>

They're reserved for chat models, but finally we did not use them for some reason

[ericzhou571]

@loofahcus I carefully check the tokenizer token map and find three speical token <|System|> <|Human|> <|Assistant|>. Will that be the speical token you plan to use to build multi round conversation? e.g.: <|Human|>repeat "this is a multi-turn conversation1" pls<|Assistant|>this is a multi-turn conversation1<|endoftext|><|Human|>repeat "this is a multi-turn conversation2" pls<|Assistant|>this is a multi-turn conversation2<|endoftext|><|Human|>repeat "this is a multi-turn conversation3" pls<|Assistant|>this is a multi-turn conversation3<|endoftext|>

They're reserved for chat models, but finally we did not use them for some reason

Would you mind to explain it a litte bit in detail? Because I am going to sft a chat model with these format. Will it lead to a poor conversation performance? Or should I use an other general chat format? e.g., CHATML?

[loofahcus]

@ericzhou571 It would not lead to a poor conversation performance once you re-initialize embed and lm_head in those positions~