Support top chinese language models

[wuxibin89]

No description provided.

[catqaq]

List of expected supported models:

• GLM
• Baichuan2
• Qwen
• InternLM
• mistral

[catqaq]

Transformers officially supports Flash Attention 2, but it only covers models like Llama. Is it necessary for us to support Flash Attention 2 for other models?

[wuxibin89]

huggingface/transformers#26350 Transformers community is adding support for more architecture.

[catqaq]

List of expected supported models:

• GLM
• Baichuan2
• Qwen
• InternLM

Done for baichuan, e770404
but requires further performance testing, and more model testing is also welcome.

Data for test:

• SFT: alpaca+alpaca_cn
• RM: hh-rlhf
• PPO: alpaca-cleaned+full-hh-rlhf+oasst1_pairwise_rlhf_reward

model	SFT	RM	PPO	Notes
Baichuan2-7B-Chat	1h	4h	71h
chatglm2-6b
Qwen-7B-Chat
internlm-chat-7b-v1_1

[catqaq]

List of expected supported models:

• GLM
• Baichuan2
• Qwen
• InternLM

Done, e770404 but requires further performance testing, and more model testing is also welcome.

Data for test:

• SFT: alpaca+alpaca_cn
• RM: hh-rlhf
• PPO: alpaca-cleaned+full-hh-rlhf+oasst1_pairwise_rlhf_reward

model SFT RM PPO Notes
Baichuan2-7B-Chat 1h 4h 71h
chatglm2-6b
Qwen-7B-Chat
internlm-chat-7b-v1_1

more test please!

[pikaqqqqqq]

Qwen-14B is being supported:

1. Different model architectures
2. Qwen cannot add extra special tokens
3. Qwen lacks eos_token setting, causing errors during dataset processing
4. Qwen supports flash_attention, but the parameter passing method is different

[jovany-wang]

What's the progress?

[Triang-jyed-driung]

Add RWKV

Would you like implementing RLHF for RWKV, here are some suggestions:

1. As a non-transformer architecture, RWKV does not use the HF transformers library.
2. RWKV uses a different tokenizer, called rwkv_vocab_v20230424, and tokenization process is based on their own tokenizer.
3. The official repositories are https://github.com/BlinkDL/RWKV-LM and https://github.com/BlinkDL/ChatRWKV. See train.py for training process.
4. RWKV contains tons of new tricks and would fail if any of these tricks is not set correctly.

[catqaq]

Add RWKV

Would you like implementing RLHF for RWKV, here are some suggestions:

1. As a non-transformer architecture, RWKV does not use the HF transformers library.
2. RWKV uses a different tokenizer, called rwkv_vocab_v20230424, and tokenization process is based on their own tokenizer.
3. The official repositories are https://github.com/BlinkDL/RWKV-LM and https://github.com/BlinkDL/ChatRWKV. See train.py for training process.
4. RWKV contains tons of new tricks and would fail if any of these tricks is not set correctly.

It seems that it is not easy to be compatible with RWKV in OpenRLHF.

[ftmtk]

I am wondering if mistral has been supported?

[catqaq]

I am wondering if mistral has been supported?

Not yet

[hijkzzz]

I am wondering if mistral has been supported?

OpenRLHF supports mistral since it's model architecture is the same as llama2.

[wxhheian]

现在qwen支持了吗

[wxhheian]

我看qwen的 特殊token <|endoftext|> 在代码里面没有体现？

[catqaq]

我看qwen的 特殊token <|endoftext|> 在代码里面没有体现？

应该是支持了，周末测测看~