Add internlm

README.md

@@ -32,7 +32,15 @@ Key features:
 [code link](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm)
 
 1. supported SFT methods: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), full(full parameter fine-tuning)
-2. supported models: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat, seqgpt-560m
+2. supported models:
+   1. qwen series: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B)
+   2. qwen-vl series: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL)
+   3. baichuan series: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat
+   4. chatglm2 series: chatglm2-6b, chatglm2-6b-32k
+   5. llama series: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat
+   6. openbuddy-llama series: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b
+   7. internlm series: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k
+   8. other: polylm-13b, seqgpt-560m
 3. supported features: quantization, ddp, model parallelism(device map), gradient checkpointing, gradient accumulation, pushing to modelscope hub, custom datasets, multimodal and agent SFT, mutli-round chat, ...
 4. supported datasets:
    1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh

README_CN.md

@@ -30,7 +30,15 @@ SWIFT（Scalable lightWeight Infrastructure for Fine-Tuning）是一个可扩展
 [code link](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm)
 
 1. 支持的SFT方法: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), 全参数微调
-2. 支持的模型: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat, seqgpt-560m
+2. 支持的模型:
+   1. qwen 系列: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B)
+   2. qwen-vl 系列: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL)
+   3. baichuan 系列: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat
+   4. chatglm2 系列: chatglm2-6b, chatglm2-6b-32k
+   5. llama 系列: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat
+   6. openbuddy-llama 系列: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b
+   7. internlm 系列: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k
+   8. other: polylm-13b, seqgpt-560m
 3. 支持的特性: 模型量化, DDP, 模型并行(device_map), gradient checkpointing, 梯度累加, 支持推送ModelScope Hub, 自定义数据集, 多模态和Agent SFT, 多轮对话, ...
 4. 支持的数据集:
    1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh

examples/pytorch/llm/README.md

@@ -16,7 +16,15 @@
 
 ## Features
 1. supported SFT methods: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), full(full parameter fine-tuning)
-2. supported models: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat, seqgpt-560m
+2. supported models:
+   1. qwen series: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B)
+   2. qwen-vl series: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL)
+   3. baichuan series: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat
+   4. chatglm2 series: chatglm2-6b, chatglm2-6b-32k
+   5. llama series: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat
+   6. openbuddy-llama series: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b
+   7. internlm series: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k
+   8. other: polylm-13b, seqgpt-560m
 3. supported features: quantization, ddp, model parallelism(device map), gradient checkpointing, gradient accumulation, pushing to modelscope hub, custom datasets, multimodal and agent SFT, mutli-round chat, ...
 4. supported datasets:
    1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh

examples/pytorch/llm/README_CN.md

@@ -17,7 +17,15 @@
 
 ## 特性
 1. 支持的SFT方法: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), 全参数微调
-2. 支持的模型: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat, seqgpt-560m
+2. 支持的模型:
+   1. qwen 系列: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B)
+   2. qwen-vl 系列: qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL)
+   3. baichuan 系列: baichuan-7b, baichuan-13b, baichuan-13b-chat, baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat
+   4. chatglm2 系列: chatglm2-6b, chatglm2-6b-32k
+   5. llama 系列: llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat
+   6. openbuddy-llama 系列: openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b
+   7. internlm 系列: internlm-7b, internlm-7b-chat, internlm-7b-chat-8k
+   8. other: polylm-13b, seqgpt-560m
 3. 支持的特性: 模型量化, DDP, 模型并行(device_map), gradient checkpointing, 梯度累加, 支持推送ModelScope Hub, 自定义数据集, 多模态和Agent SFT, 多轮对话, ...
 4. 支持的数据集:
    1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en, cmnli-zh

examples/pytorch/llm/scripts/internlm_7b_chat/lora_ddp/infer.sh

@@ -0,0 +1,13 @@
+CUDA_VISIBLE_DEVICES=0 \
+python src/llm_infer.py \
+    --model_type internlm-7b-chat \
+    --sft_type lora \
+    --template_type internlm \
+    --dtype bf16 \
+    --ckpt_dir "runs/internlm-7b-chat/vx_xxx/checkpoint-xxx" \
+    --eval_human true \
+    --max_new_tokens 1024 \
+    --temperature 0.9 \
+    --top_k 50 \
+    --top_p 0.9 \
+    --do_sample true \

examples/pytorch/llm/scripts/internlm_7b_chat/lora_ddp/sft.sh

@@ -0,0 +1,35 @@
+nproc_per_node=2
+CUDA_VISIBLE_DEVICES=0,1 \
+torchrun \
+    --nproc_per_node=$nproc_per_node \
+    --master_port 29500 \
+    src/llm_sft.py \
+    --model_type internlm-7b-chat \
+    --sft_type lora \
+    --template_type internlm \
+    --dtype bf16 \
+    --output_dir runs \
+    --ddp_backend nccl \
+    --dataset alpaca-en,alpaca-zh \
+    --dataset_sample 20000 \
+    --num_train_epochs 1 \
+    --max_length 1024 \
+    --lora_rank 8 \
+    --lora_alpha 32 \
+    --lora_dropout_p 0.05 \
+    --lora_target_modules q_proj k_proj v_proj \
+    --gradient_checkpointing false \
+    --batch_size 1 \
+    --weight_decay 0. \
+    --learning_rate 1e-4 \
+    --gradient_accumulation_steps $(expr 16 / $nproc_per_node) \
+    --max_grad_norm 0.5 \
+    --warmup_ratio 0.03 \
+    --eval_steps 100 \
+    --save_steps 100 \
+    --save_total_limit 2 \
+    --logging_steps 10 \
+    --push_to_hub false \
+    --hub_model_id internlm-7b-lora \
+    --hub_private_repo true \
+    --hub_token 'your-sdk-token' \

examples/pytorch/llm/scripts/qwen_7b/lora_ddp/infer.sh

@@ -6,6 +6,7 @@ python src/llm_infer.py \
     --dtype bf16 \
     --ckpt_dir "runs/qwen-7b/vx_xxx/checkpoint-xxx" \
     --eval_human true \
+    --use_flash_attn true \
     --max_new_tokens 1024 \
     --temperature 0.9 \
     --top_k 50 \

examples/pytorch/llm/scripts/qwen_7b/lora_ddp/sft.sh

@@ -29,7 +29,7 @@ torchrun \
     --save_steps 100 \
     --save_total_limit 2 \
     --logging_steps 10 \
-    --use_flash_attn false \
+    --use_flash_attn true \
     --push_to_hub false \
     --hub_model_id qwen-7b-lora \
     --hub_private_repo true \

examples/pytorch/llm/scripts/qwen_vl/lora_ddp/infer.sh

@@ -8,6 +8,7 @@ python src/llm_infer.py \
     --eval_human false \
     --dataset coco-en \
     --dataset_sample 20000 \
+    --use_flash_attn true \
     --max_new_tokens 1024 \
     --temperature 0.9 \
     --top_k 50 \

examples/pytorch/llm/scripts/qwen_vl/lora_ddp/sft.sh

@@ -29,7 +29,7 @@ torchrun \
     --save_steps 100 \
     --save_total_limit 2 \
     --logging_steps 10 \
-    --use_flash_attn false \
+    --use_flash_attn true \
     --push_to_hub false \
     --hub_model_id qwen-vl-lora \
     --hub_private_repo true \

examples/pytorch/llm/src/utils/dataset.py

@@ -290,8 +290,7 @@ def get_gpt4all_en_dataset() -> HfDataset:
 
 def _preprocess_nli(dataset: HfDataset) -> HfDataset:
     cls_mapping = ['neutral', 'entailment', 'contradiction']
-    prompt = """
-Input:
+    prompt = """Input:
     Sentence1: {sentence1}
     Sentence2: {sentence2}
 Classification: neutral, entailment, contradiction

examples/pytorch/llm/src/utils/model.py

@@ -172,6 +172,7 @@ class LoRATM(NamedTuple):
     qwen = ['c_attn']
     polylm = ['c_attn']
     bloom = ['query_key_value']
+    internlm = ['q_proj', 'k_proj', 'v_proj']
 
 
 # Model Home: 'https://modelscope.cn/models/{model_id}/summary'
@@ -327,7 +328,24 @@ class LoRATM(NamedTuple):
         'revision': 'v1.0.1',
         'template': 'default-generation',
         'lora_TM': LoRATM.bloom,
-    }
+    },
+    'internlm-7b': {
+        'model_id': 'Shanghai_AI_Laboratory/internlm-7b',
+        'revision': 'v1.0.0',
+        'lora_TM': LoRATM.internlm,
+    },
+    'internlm-7b-chat': {
+        'model_id': 'Shanghai_AI_Laboratory/internlm-chat-7b-v1_1',
+        'revision': 'v1.0.0',
+        'template': 'internlm',
+        'lora_TM': LoRATM.internlm,
+    },
+    'internlm-7b-chat-8k': {
+        'model_id': 'Shanghai_AI_Laboratory/internlm-chat-7b-8k',
+        'revision': 'v1.0.0',
+        'template': 'internlm',
+        'lora_TM': LoRATM.internlm,
+    },
 }
 
 

examples/pytorch/llm/src/utils/preprocess.py

@@ -19,14 +19,11 @@
         'suffix': [['eos_token_id']],
     },
     'chatml': {
-        'prefix': [['im_start_id'], 'system\n{{system}}', ['im_end_id'], '\n'],
-        'prompt': [['im_start_id'], 'user\n{{query}}', ['im_end_id'], '\n',
-                   ['im_start_id'], 'assistant\n'],
-        'chat_sep': [
-            ['im_end_id'],
-            '\n',
-        ],
-        'suffix': [['im_end_id'], ['eod_id']],
+        'prefix': ['<|im_start|>system\n{{system}}<|im_end|>\n'],
+        'prompt':
+        ['<|im_start|>user\n{{query}}<|im_end|>\n<|im_start|>assistant\n'],
+        'chat_sep': ['<|im_end|>\n'],
+        'suffix': ['<|im_end|><|endoftext|>'],
     },
     'baichuan': {
         'prefix': [],
@@ -52,6 +49,12 @@
         'prompt': ['User: {{query}}\nAssistant: '],
         'chat_sep': ['\n'],
         'suffix': [['eos_token_id']],
+    },
+    'internlm': {
+        'prefix': ['<s>'],
+        'prompt': ['<|User|>:{{query}}<eoh>\n<|Bot|>:'],
+        'chat_sep': ['<eoa>\n'],
+        'suffix': ['<eoa></s>'],
     }
 }
 Context = Union[str, List[int]]