diff --git a/README.md b/README.md index a5c8921e74..e5d72aa36c 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,15 @@ Key features: [code link](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm) 1. supported SFT methods: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), full(full parameter fine-tuning) -2. supported models: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat, seqgpt-560m +2. supported models: + 1. qwen series: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B) + 2. qwen-vl series: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL) + 3. baichuan series: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat + 4. chatglm2 series: chatglm2-6b, chatglm2-6b-32k + 5. llama series: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat + 6. openbuddy-llama series: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b + 7. internlm series: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k + 8. other: polylm-13b, seqgpt-560m 3. supported features: quantization, ddp, model parallelism(device map), gradient checkpointing, gradient accumulation, pushing to modelscope hub, custom datasets, multimodal and agent SFT, mutli-round chat, ... 4. supported datasets: 1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh diff --git a/README_CN.md b/README_CN.md index 7fb9a80aa1..878c90ad31 100644 --- a/README_CN.md +++ b/README_CN.md @@ -30,7 +30,15 @@ SWIFT（Scalable lightWeight Infrastructure for Fine-Tuning）是一个可扩展 [code link](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm) 1. 支持的SFT方法: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), 全参数微调 -2. 支持的模型: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat, seqgpt-560m +2. 支持的模型: + 1. qwen 系列: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B) + 2. qwen-vl 系列: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL) + 3. baichuan 系列: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat + 4. chatglm2 系列: chatglm2-6b, chatglm2-6b-32k + 5. llama 系列: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat + 6. openbuddy-llama 系列: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b + 7. internlm 系列: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k + 8. other: polylm-13b, seqgpt-560m 3. 支持的特性: 模型量化, DDP, 模型并行(device_map), gradient checkpointing, 梯度累加, 支持推送ModelScope Hub, 自定义数据集, 多模态和Agent SFT, 多轮对话, ... 4. 支持的数据集: 1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh diff --git a/examples/pytorch/llm/README.md b/examples/pytorch/llm/README.md index 056233187e..b68499b6ab 100644 --- a/examples/pytorch/llm/README.md +++ b/examples/pytorch/llm/README.md @@ -16,7 +16,15 @@ ## Features 1. supported SFT methods: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), full(full parameter fine-tuning) -2. supported models: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat, seqgpt-560m +2. supported models: + 1. qwen series: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B) + 2. qwen-vl series: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL) + 3. baichuan series: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat + 4. chatglm2 series: chatglm2-6b, chatglm2-6b-32k + 5. llama series: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat + 6. openbuddy-llama series: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b + 7. internlm series: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k + 8. other: polylm-13b, seqgpt-560m 3. supported features: quantization, ddp, model parallelism(device map), gradient checkpointing, gradient accumulation, pushing to modelscope hub, custom datasets, multimodal and agent SFT, mutli-round chat, ... 4. supported datasets: 1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh diff --git a/examples/pytorch/llm/README_CN.md b/examples/pytorch/llm/README_CN.md index 40f57d5ca0..2d75189891 100644 --- a/examples/pytorch/llm/README_CN.md +++ b/examples/pytorch/llm/README_CN.md @@ -17,7 +17,15 @@ ## 特性 1. 支持的SFT方法: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), 全参数微调 -2. 支持的模型: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat, seqgpt-560m +2. 支持的模型: + 1. qwen 系列: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B) + 2. qwen-vl 系列: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL) + 3. baichuan 系列: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat + 4. chatglm2 系列: chatglm2-6b, chatglm2-6b-32k + 5. llama 系列: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat + 6. openbuddy-llama 系列: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b + 7. internlm 系列: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k + 8. other: polylm-13b, seqgpt-560m 3. 支持的特性: 模型量化, DDP, 模型并行(device_map), gradient checkpointing, 梯度累加, 支持推送ModelScope Hub, 自定义数据集, 多模态和Agent SFT, 多轮对话, ... 4. 支持的数据集: 1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh diff --git a/examples/pytorch/llm/scripts/internlm_7b_chat/lora_ddp/infer.sh b/examples/pytorch/llm/scripts/internlm_7b_chat/lora_ddp/infer.sh new file mode 100644 index 0000000000..be9c767529 --- /dev/null +++ b/examples/pytorch/llm/scripts/internlm_7b_chat/lora_ddp/infer.sh @@ -0,0 +1,13 @@ +CUDA_VISIBLE_DEVICES=0 \ +python src/llm_infer.py \ + --model_type internlm-7b-chat \ + --sft_type lora \ + --template_type internlm \ + --dtype bf16 \ + --ckpt_dir "runs/internlm-7b-chat/vx_xxx/checkpoint-xxx" \ + --eval_human true \ + --max_new_tokens 1024 \ + --temperature 0.9 \ + --top_k 50 \ + --top_p 0.9 \ + --do_sample true \ diff --git a/examples/pytorch/llm/scripts/internlm_7b_chat/lora_ddp/sft.sh b/examples/pytorch/llm/scripts/internlm_7b_chat/lora_ddp/sft.sh new file mode 100644 index 0000000000..3fcb0883a9 --- /dev/null +++ b/examples/pytorch/llm/scripts/internlm_7b_chat/lora_ddp/sft.sh @@ -0,0 +1,35 @@ +nproc_per_node=2 +CUDA_VISIBLE_DEVICES=0,1 \ +torchrun \ + --nproc_per_node=$nproc_per_node \ + --master_port 29500 \ + src/llm_sft.py \ + --model_type internlm-7b-chat \ + --sft_type lora \ + --template_type internlm \ + --dtype bf16 \ + --output_dir runs \ + --ddp_backend nccl \ + --dataset alpaca-en,alpaca-zh \ + --dataset_sample 20000 \ + --num_train_epochs 1 \ + --max_length 1024 \ + --lora_rank 8 \ + --lora_alpha 32 \ + --lora_dropout_p 0.05 \ + --lora_target_modules q_proj k_proj v_proj \ + --gradient_checkpointing false \ + --batch_size 1 \ + --weight_decay 0. \ + --learning_rate 1e-4 \ + --gradient_accumulation_steps $(expr 16 / $nproc_per_node) \ + --max_grad_norm 0.5 \ + --warmup_ratio 0.03 \ + --eval_steps 100 \ + --save_steps 100 \ + --save_total_limit 2 \ + --logging_steps 10 \ + --push_to_hub false \ + --hub_model_id internlm-7b-lora \ + --hub_private_repo true \ + --hub_token 'your-sdk-token' \ diff --git a/examples/pytorch/llm/scripts/qwen_7b/lora_ddp/infer.sh b/examples/pytorch/llm/scripts/qwen_7b/lora_ddp/infer.sh index c5f9825c52..dfe1605a0d 100644 --- a/examples/pytorch/llm/scripts/qwen_7b/lora_ddp/infer.sh +++ b/examples/pytorch/llm/scripts/qwen_7b/lora_ddp/infer.sh @@ -6,6 +6,7 @@ python src/llm_infer.py \ --dtype bf16 \ --ckpt_dir "runs/qwen-7b/vx_xxx/checkpoint-xxx" \ --eval_human true \ + --use_flash_attn true \ --max_new_tokens 1024 \ --temperature 0.9 \ --top_k 50 \ diff --git a/examples/pytorch/llm/scripts/qwen_7b/lora_ddp/sft.sh b/examples/pytorch/llm/scripts/qwen_7b/lora_ddp/sft.sh index 5c2b8c00ae..522d07435c 100644 --- a/examples/pytorch/llm/scripts/qwen_7b/lora_ddp/sft.sh +++ b/examples/pytorch/llm/scripts/qwen_7b/lora_ddp/sft.sh @@ -29,7 +29,7 @@ torchrun \ --save_steps 100 \ --save_total_limit 2 \ --logging_steps 10 \ - --use_flash_attn false \ + --use_flash_attn true \ --push_to_hub false \ --hub_model_id qwen-7b-lora \ --hub_private_repo true \ diff --git a/examples/pytorch/llm/scripts/qwen_vl/lora_ddp/infer.sh b/examples/pytorch/llm/scripts/qwen_vl/lora_ddp/infer.sh index d35f4bb5bb..f3f6732f5a 100644 --- a/examples/pytorch/llm/scripts/qwen_vl/lora_ddp/infer.sh +++ b/examples/pytorch/llm/scripts/qwen_vl/lora_ddp/infer.sh @@ -8,6 +8,7 @@ python src/llm_infer.py \ --eval_human false \ --dataset coco-en \ --dataset_sample 20000 \ + --use_flash_attn true \ --max_new_tokens 1024 \ --temperature 0.9 \ --top_k 50 \ diff --git a/examples/pytorch/llm/scripts/qwen_vl/lora_ddp/sft.sh b/examples/pytorch/llm/scripts/qwen_vl/lora_ddp/sft.sh index 5d312d3224..dc28206a09 100644 --- a/examples/pytorch/llm/scripts/qwen_vl/lora_ddp/sft.sh +++ b/examples/pytorch/llm/scripts/qwen_vl/lora_ddp/sft.sh @@ -29,7 +29,7 @@ torchrun \ --save_steps 100 \ --save_total_limit 2 \ --logging_steps 10 \ - --use_flash_attn false \ + --use_flash_attn true \ --push_to_hub false \ --hub_model_id qwen-vl-lora \ --hub_private_repo true \ diff --git a/examples/pytorch/llm/src/utils/dataset.py b/examples/pytorch/llm/src/utils/dataset.py index 4184c9f948..18c9c62e44 100644 --- a/examples/pytorch/llm/src/utils/dataset.py +++ b/examples/pytorch/llm/src/utils/dataset.py @@ -290,8 +290,7 @@ def get_gpt4all_en_dataset() -> HfDataset: def _preprocess_nli(dataset: HfDataset) -> HfDataset: cls_mapping = ['neutral', 'entailment', 'contradiction'] - prompt = """ -Input: + prompt = """Input: Sentence1: {sentence1} Sentence2: {sentence2} Classification: neutral, entailment, contradiction diff --git a/examples/pytorch/llm/src/utils/model.py b/examples/pytorch/llm/src/utils/model.py index 0765640ddc..6b456fe54e 100644 --- a/examples/pytorch/llm/src/utils/model.py +++ b/examples/pytorch/llm/src/utils/model.py @@ -172,6 +172,7 @@ class LoRATM(NamedTuple): qwen = ['c_attn'] polylm = ['c_attn'] bloom = ['query_key_value'] + internlm = ['q_proj', 'k_proj', 'v_proj'] # Model Home: 'https://modelscope.cn/models/{model_id}/summary' @@ -327,7 +328,24 @@ class LoRATM(NamedTuple): 'revision': 'v1.0.1', 'template': 'default-generation', 'lora_TM': LoRATM.bloom, - } + }, + 'internlm-7b': { + 'model_id': 'Shanghai_AI_Laboratory/internlm-7b', + 'revision': 'v1.0.0', + 'lora_TM': LoRATM.internlm, + }, + 'internlm-7b-chat': { + 'model_id': 'Shanghai_AI_Laboratory/internlm-chat-7b-v1_1', + 'revision': 'v1.0.0', + 'template': 'internlm', + 'lora_TM': LoRATM.internlm, + }, + 'internlm-7b-chat-8k': { + 'model_id': 'Shanghai_AI_Laboratory/internlm-chat-7b-8k', + 'revision': 'v1.0.0', + 'template': 'internlm', + 'lora_TM': LoRATM.internlm, + }, } diff --git a/examples/pytorch/llm/src/utils/preprocess.py b/examples/pytorch/llm/src/utils/preprocess.py index fb30ef6cdc..4d26613570 100644 --- a/examples/pytorch/llm/src/utils/preprocess.py +++ b/examples/pytorch/llm/src/utils/preprocess.py @@ -19,14 +19,11 @@ 'suffix': [['eos_token_id']], }, 'chatml': { - 'prefix': [['im_start_id'], 'system\n{{system}}', ['im_end_id'], '\n'], - 'prompt': [['im_start_id'], 'user\n{{query}}', ['im_end_id'], '\n', - ['im_start_id'], 'assistant\n'], - 'chat_sep': [ - ['im_end_id'], - '\n', - ], - 'suffix': [['im_end_id'], ['eod_id']], + 'prefix': ['<|im_start|>system\n{{system}}<|im_end|>\n'], + 'prompt': + ['<|im_start|>user\n{{query}}<|im_end|>\n<|im_start|>assistant\n'], + 'chat_sep': ['<|im_end|>\n'], + 'suffix': ['<|im_end|><|endoftext|>'], }, 'baichuan': { 'prefix': [], @@ -52,6 +49,12 @@ 'prompt': ['User: {{query}}\nAssistant: '], 'chat_sep': ['\n'], 'suffix': [['eos_token_id']], + }, + 'internlm': { + 'prefix': [''], + 'prompt': ['<|User|>:{{query}}\n<|Bot|>:'], + 'chat_sep': ['\n'], + 'suffix': [''], } } Context = Union[str, List[int]]