Refactor chat template and support accurate name matching. #1216
Conversation
|
Need update after #1168 merged |
|
#1168 has been merged |
Conflicts: lmdeploy/model.py lmdeploy/turbomind/utils.py
lmdeploy/model.py
Outdated
| eoh='\n', | ||
| assistant='ASSISTANT: ', | ||
| eoa='</s>', | ||
| separator='\n', |
There was a problem hiding this comment.
vicuna decorate之后跟之前的不一样，可以看一下
There was a problem hiding this comment.
是有点奇怪，FastChat 文档里面的模板和代码效果略有差异。文档里效果是这样的：
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
USER: Hello!
ASSISTANT: Hello!</s>
USER: How are you?
ASSISTANT: I am good.</s>
然后代码效果这样的：
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Hello! ASSISTANT: Hi!</s>USER: How are you? ASSISTANT:
lmdeploy/model.py
Outdated
| box_map = dict(user=self.user, | ||
| assistant=self.assistant, | ||
| system=self.system) | ||
| eox_map = dict(user=self.eoh, | ||
| assistant=self.eoa + self.separator, | ||
| system=self.eosys) | ||
| ret = '' | ||
| for message in messages: | ||
| role = message['role'] | ||
| content = message['content'] | ||
| ret += f'{box_map[role]}{content}{eox_map[role]}' | ||
| ret += f'{self.assistant}' |
There was a problem hiding this comment.
messages2prompt 是说messages里面没有system的话，就不加了是么？这个之前貌似默认是有的。
There was a problem hiding this comment.
对，这个行为被BC了（internlm 的模型除外），需要讨论下。之前如果用户不传 meta_instrucion，会用默认的。现在只看用户传的，如果用户没传就没在用。
There was a problem hiding this comment.
确实有些纠结。这里修改的出发点是，可以简化些逻辑，不用translate_messages，对吧。
There was a problem hiding this comment.
确实有些纠结。这里修改的出发点是，可以简化些逻辑，不用translate_messages，对吧。
对，其实我们是不知道ChatGPT内部的行为是什么。不过像 FastChat 似乎内部是用了缺省值的，如果用户没传的话。
|
|
Conflicts: lmdeploy/cli/cli.py
|
I am doing BC-breaking test, comparing the results between v0.2.5 and this PR. They are different prompt = 'hi, how are you'
from lmdeploy.model import MODELS
chat_templates = [
'internlm', 'llama', 'base',
'wizardlm', 'vicuna',
'internlm-chat', 'internlm-chat-7b', 'internlm-chat-20b', 'internlm-chat-7b-8k',
'internlm2-chat', 'internlm2-chat-1_8b', 'internlm2-chat-7b', 'internlm2-chat-20b',
'baichuan-7b',
'baichuan2-7b',
'llama2', 'llama-2', 'llama-2-chat',
'qwen-7b', 'qwen-14b',
'codellama',
'falcon',
'chatglm2-6b',
'solar', 'solar-70b',
'ultracm',
'ultralm',
'yi', 'yi-chat', 'yi-200k', 'yi-34b',
'Mistral-7B-Instruct', 'Mixtral-8x7B-Instruct',
'gemma',
'deepseek-chat'
]
for _template in chat_templates:
print(f'---------{_template}---------')
model = MODELS.get(_template)()
print(model.get_prompt(prompt)) |
|
还有个问题，base 模型的匹配，现在基本都会匹配不到名字。 |
AllentDan
force-pushed
the
refactor-model
branch
from
a9adb0f
to
c7d83fd
Compare
from_config,decorate_prompt,_translate_messages, andsampling_paramfrom the chat template.