update tokenizer process.

[shibing624]

1. support llama2, baichuan, bloom, chatglm model data tokenize
2. make the code more readable

Why are these changes needed?

Related issue number (if applicable)

Checks

• I've run format.sh to lint the changes in this PR.
• I've included any doc changes needed.
• I've made sure the relevant tests are passing (if applicable).

[merrymercy]

Could you double-check that the output of this line (

FastChat/fastchat/train/train.py

Lines 140 to 143 in 820de33

	if False: # Inspect and check the correctness of masking
	z = target.clone()
	z = torch.where(z == IGNORE_TOKEN_ID, tokenizer.unk_token_id, z)
	rank0_print(tokenizer.decode(z))

) is exactly the same before and after your change?
You can see more discussions and test cases at #1894 (comment)

[shibing624]

i double chekced, it is right with llama2.

[merrymercy]

@shibing624 Thanks! It is merged.

[merrymercy]

@shibing624 The first token is for BOS. I found BOS is dropped after your PR. Could you fix this?

[merrymercy]

I reverted this PR because of the BOS problem (#2138 (comment)). Please fix it and submit a new one.
This part is very error-prone. Please make sure input_ids and targets keep exactly the same for Llama models after your PR.

[shibing624]

yes, the tokenizer process with split and find multi round eos_token is very error-prone. so i rewrite the logic of get dialogs from preprocess, then just need tokenize prompt and answer, the code is changed in my repo: https://github.com/shibing624/MedicalGPT/blob/main/supervised_finetuning.py#L653