AttributeError: 'ChatGLMTokenizer' object has no attribute 'tokenizer'

[digitalghost]

问题描述 / Problem Description
从0.25开始使用，对于配置都比较清楚了，今天更新了0.26之后，发生了以下异常。
ChatGLM2和M3e-base用的都是放在本地目录里面的文件，是不是huggingface的什么api发生了变化，导致的这个问题？

2023-10-23 03:55:19 | INFO | model_worker | Loading the model ['chatglm2-6b'] on worker 4b962e02 ...
2023-10-23 03:55:19 | ERROR | stderr | Process model_worker - chatglm2-6b:
2023-10-23 03:55:19 | ERROR | stderr | Traceback (most recent call last):
2023-10-23 03:55:19 | INFO | model_worker | Register to controller
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
2023-10-23 03:55:19 | ERROR | stderr | self.run()
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/multiprocessing/process.py", line 108, in run
2023-10-23 03:55:19 | ERROR | stderr | self._target(*self._args, **self._kwargs)
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/projects/Langchain-Chatchat-master/startup.py", line 382, in run_model_worker
2023-10-23 03:55:19 | ERROR | stderr | app = create_model_worker_app(log_level=log_level, **kwargs)
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/projects/Langchain-Chatchat-master/startup.py", line 210, in create_model_worker_app
2023-10-23 03:55:19 | ERROR | stderr | worker = ModelWorker(
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/site-packages/fastchat/serve/model_worker.py", line 74, in init
2023-10-23 03:55:19 | ERROR | stderr | self.model, self.tokenizer = load_model(
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/site-packages/fastchat/model/model_adapter.py", line 294, in load_model
2023-10-23 03:55:19 | ERROR | stderr | model, tokenizer = adapter.load_model(model_path, kwargs)
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/site-packages/fastchat/model/model_adapter.py", line 716, in load_model
2023-10-23 03:55:19 | ERROR | stderr | tokenizer = AutoTokenizer.from_pretrained(
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 738, in from_pretrained
2023-10-23 03:55:19 | ERROR | stderr | return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2017, in from_pretrained
2023-10-23 03:55:19 | ERROR | stderr | return cls._from_pretrained(
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2249, in _from_pretrained
2023-10-23 03:55:19 | ERROR | stderr | tokenizer = cls(*init_inputs, **init_kwargs)
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/.cache/huggingface/modules/transformers_modules/chatglm2-6b/tokenization_chatglm.py", line 69, in init
2023-10-23 03:55:19 | ERROR | stderr | super().init(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs)
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 367, in init
2023-10-23 03:55:19 | ERROR | stderr | self._add_tokens(
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/anaconda3/envs/langchain-0.26/lib/python3.11/site-packages/transformers/tokenization_utils.py", line 467, in _add_tokens
2023-10-23 03:55:19 | ERROR | stderr | current_vocab = self.get_vocab().copy()
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/.cache/huggingface/modules/transformers_modules/chatglm2-6b/tokenization_chatglm.py", line 112, in get_vocab
2023-10-23 03:55:19 | ERROR | stderr | vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | File "/home/alan/.cache/huggingface/modules/transformers_modules/chatglm2-6b/tokenization_chatglm.py", line 108, in vocab_size
2023-10-23 03:55:19 | ERROR | stderr | return self.tokenizer.n_words
2023-10-23 03:55:19 | ERROR | stderr | ^^^^^^^^^^^^^^
2023-10-23 03:55:19 | ERROR | stderr | AttributeError: 'ChatGLMTokenizer' object has no attribute 'tokenizer'

复现问题的步骤 / Steps to Reproduce
启动 python startup.py -a

[digitalghost]

解决了，去huggingface更新一下
tokenization_chatglm.py
这个文件就可以了

[ztfoolish]

请问具体怎么更新？我是把模型下载到本地的

[zhaoqianyue]

请问具体怎么更新？我是把模型下载到本地的

直接下载啊

[ztfoolish]

好的，谢谢～我以为是要改代码。hf现在挂了梯子也经常访问不了，难搞:(

[badhard]

降低 transformers 版本就可以跑起来了
pip uninstall transformers
pip install -i https://pypi.tuna.tsinghua.edu.cn/simple transformers==4.33.2

[watreyoung]

最新版本的文件在tokenization_chatglm.py，替换掉可以使用。