Remove duplicated bos_token for CodeLlama

[alealv]

Fixes #1565

[rasbt]

I just saw your issue and was about to suggest a PR. Thanks for already doing that! This is really appreciated!

The update looks good to me, but I am tagging @Andrei-Aksionov for a second pair of eyes (I think he saw something similar with other models)

[Andrei-Aksionov]

Hello @alealv

Thanks for the PR.

I believe that prompt template was chosen since it's the one that is used in HF Transformers and, I guess, it was provided by the author of the model.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-Instruct-hf")
messages = [
    {"role": "user", "content": "List all presidents of the USA."},
]
tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

'<s>[INST] List all presidents of the USA. [/INST]'

Although I agree that it's weird to hard-code bos token in a prompt when in the tokenizer_config.json add_bos_token is set to True.

Nevertheless, this issue uncovered other problems:

1. .encode should deal with such cases.
   Update: the CodeLlaMA tokenizer return a bos token no matter what (before if-else statement where we add bos manually). So the check isn't applied.
2. HF AutoTokenizer doesn't return doubled bos token, so a test should have failed. But it didn't.
   Update: everything is correct. It's just tokenizer.apply_chat_template does additional checks if bos is duplicated. Ours code output is aligned with AutoTokenizer.

Overall, I think we can merge the PR and in the meanwhile I'll investigate these two issues.

[rasbt]

Thanks for feedback @Andrei-Aksionov . Let's merge it and then investigate the 2 other points separately.