chatglm3-6b-base AttributeError: can't set attribute

[xiaohangguo]

尝试解析chatglm3-base全量微调完的模型，然后解析了一下，得到了这个错误

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("/public/home/lvshuhang/xtuner/work_dirs/chatglm3_6b_base_full_alpaca_e3/epoch3_v2_ckpt", trust_remote_code=True)
model = AutoModel.from_pretrained("/public/home/lvshuhang/xtuner/work_dirs/chatglm3_6b_base_full_alpaca_e3/epoch3_v2_ckpt", trust_remote_code=True).half().cuda()

inputs = tokenizer(["今天天气真不错"], return_tensors="pt").to('cuda')
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0].tolist()))

(lmflow) [lvshuhang@node61 xtuner]$ python Tianchi-LLM-retrieval/demo.py
Traceback (most recent call last):
  File "/public/home/lvshuhang/xtuner/Tianchi-LLM-retrieval/demo.py", line 2, in <module>
    tokenizer = AutoTokenizer.from_pretrained("/public/home/lvshuhang/xtuner/work_dirs/chatglm3_6b_base_full_alpaca_e3/epoch3_v2_ckpt/", trust_remote_code=True)
  File "/public/home/lvshuhang/miniconda3/envs/lmflow/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 755, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/public/home/lvshuhang/miniconda3/envs/lmflow/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2024, in from_pretrained
    return cls._from_pretrained(
  File "/public/home/lvshuhang/miniconda3/envs/lmflow/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2256, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/public/home/lvshuhang/.cache/huggingface/modules/transformers_modules/tokenization_chatglm.py", line 108, in __init__
    super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces,
  File "/public/home/lvshuhang/miniconda3/envs/lmflow/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 363, in __init__
    super().__init__(**kwargs)
  File "/public/home/lvshuhang/miniconda3/envs/lmflow/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1604, in __init__
    super().__init__(**kwargs)
  File "/public/home/lvshuhang/miniconda3/envs/lmflow/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 861, in __init__
    setattr(self, key, value)
AttributeError: can't set attribute

tokenization_chatglm.py 貌似是这个文件引发的，好像是和transformers的名字重了？
这个可能跟这个问题一样
huggingface/transformers#25194

[LZHgrla]

建议将相关问题提交至chatglm社区，在下游仓库（如transformers、xtuner）无法进行完全的解决。
一种可能的解决方案（未验证）是直接使用原版tokenzier

tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm3-6b-base", trust_remote_code=True)