generalized chat sft prompt

[yidong72]

What does this PR do ?

In this PR, it genialized the chat SFT dataset that it can use customized turn start/end tokens by using chat_prompt_tokens config. e.g.

    chat_prompt_tokens:  # special tokens for the chat prompts, a dictionary of {token_type: token}. note that some tokenizer may combine the characters at the junction between {end_of_turn}{turn_start}. e.g. '<im end><im start>', the '><' sometimes is merged to be a single token. This is not supported, try to avoid
      system_turn_start: '<extra_id_0>'
      turn_start: '<extra_id_1>'
      label_start: '<extra_id_2>'
      end_of_turn: "\x0A"  # \0x0A is '\n'
      end_of_name: "\x0A"  # \0x0A is '\n'

after this change, the LM is not required to have "extra_id" special tokens any more to use chat SFT dataset. In this PR, also expanded the unit test to cover more LM tokenizers.

Another feature added is to overwrite the prompt_template config with the chat prompt format.

[Zhilin123]

LGTM, minor code style issues

[Zhilin123]

LGTM

[aklife97]

can we do a check to see if the tokens in special_tokens are tokenizer's special tokens or not? If not (the case with llama), can we just throw a warning that we'll use text as turn tokens which might cause incorrect merging

[yidong72]

I have an assert in the code

assert torch.equal(torch.tensor(target[:header_len]), torch.tensor(header_tokens))

which will throw an exception if the token merge happens.

[aklife97]

that is different, the token merge can still happen during multi-turn

what I mean is that if the turn tokens are not special tokens, we just say that there might be an error possible

[yidong72]

The header_len stops at the "end_of_turn". The next token is "turn_start". If the merge happens this assert will catch it. The multiple turn has the same thing. each turn ends with "end_of_turn" and the next token is "turn_start". So this one is enough to catch it.

Also I don't see the point of just giving a warning which doesn't help the user at all.

[aklife97]

LGTM, thanks!