chatglm2 support

[elven2016]

Describe the bug

chatglm2-6b load successfully but canot chat，input message response nothing，and the terminal show the following errors

Traceback (most recent call last):
File "/ssd_data01/text-generation-webui/modules/callbacks.py", line 73, in gentask
ret = self.mfunc(callback=_callback, **self.kwargs)
File "/ssd_data01/text-generation-webui/modules/text_generation.py", line 263, in generate_with_callback
shared.model.generate(**kwargs)
File "/home/elven/miniconda3/envs/tgweb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/elven/miniconda3/envs/tgweb/lib/python3.10/site-packages/transformers/generation/utils.py", line 1285, in generate
eos_token_id = eos_token_id[0]
IndexError: list index out of range

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

load the model and in chat message mode ，input the message in the chabox，response nothing

Screenshot

Logs

Traceback (most recent call last):
  File "/ssd_data01/text-generation-webui/modules/callbacks.py", line 73, in gentask
    ret = self.mfunc(callback=_callback, **self.kwargs)
  File "/ssd_data01/text-generation-webui/modules/text_generation.py", line 263, in generate_with_callback
    shared.model.generate(**kwargs)
  File "/home/elven/miniconda3/envs/tgweb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/elven/miniconda3/envs/tgweb/lib/python3.10/site-packages/transformers/generation/utils.py", line 1285, in generate
    eos_token_id = eos_token_id[0]
IndexError: list index out of range

System Info

ubuntu22.04 LTS
python3.10.10

[junphine]

on tokenization_chatglm.py

line：78 add self._eos_token=''

[junphine]

line：78 add self._eos_token='<eos>'

[elven2016]

line：78 add self._eos_token=''

感谢回复。添加这个之后能回复了，但是回复内容会胡言乱语，并且会不终止，一直发个不停，而且还是没有问的问题，

[upbit]

感谢回复。添加这个之后能回复了，但是回复内容会胡言乱语，并且会不终止，一直发个不停，而且还是没有问的问题

The same question as the one asked in the previous message, comparing ChatGLM2's demo, will output "// Another topic" in the answer. What is the reason for this? Did ChatGLM2 use special ending symbols?

ChatGLM2 web_demo

WebUI

Any ideas?

[upbit]

Found the reason(thanks lejunzhu): Similar to LLaM's quick fix, it needs to be manually set the eos/bos/pad token id:

diff --git a/modules/models.py b/modules/models.py
index 4b47e64..dea9e73 100644
--- a/modules/models.py
+++ b/modules/models.py
@@ -116,6 +116,17 @@ def load_tokenizer(model_name, model):
         if path_to_model.exists():
             tokenizer = AutoTokenizer.from_pretrained(path_to_model, trust_remote_code=shared.args.trust_remote_code)

+        if 'chatglm' in model_name.lower():
+            try:
+                tokenizer.eos_token_id = 2
+            except:
+                pass
+            try:
+                tokenizer.bos_token_id = 1
+                tokenizer.pad_token_id = 0
+            except:
+                pass
+
     return tokenizer

But I don't know why my tokenizer.eos_token_id has a value here, setting it directly will cause error, so I divided it into 2 try..catch

[elven2016]

thanks your reply， but I set it according to your code change，the same error again，no response in chat windows
Traceback (most recent call last):
File "/ssd_data01/text-generation-webui/modules/callbacks.py", line 55, in gentask
ret = self.mfunc(callback=_callback, *args, **self.kwargs)
File "/ssd_data01/text-generation-webui/modules/text_generation.py", line 289, in generate_with_callback
shared.model.generate(**kwargs)
File "/home/elven/miniconda3/envs/tgweb/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/elven/miniconda3/envs/tgweb/lib/python3.10/site-packages/transformers/generation/utils.py", line 1285, in generate
eos_token_id = eos_token_id[0]
IndexError: list index out of range

[elven2016]

现在已经可以正常回复了 ，我稍微改了一些上面Zhouhao的建议修改方式， def load_tokenizer(model_name, model): 这个函数增加的代码如下图：118-124行是新增的代码

[github-actions]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.