Baichuan2-7B-Chat执行推理报错TypeError: sdp_kernel() got an unexpected keyword argument 'enable_mem_efficient'

[liding1992]

我的环境信息：

centos 7.9
python 3.10.9
cuda 11.7
transformers 4.33.1
torch 1.13.1+cu117

推理脚本：

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation.utils import GenerationConfig

model_path='/home/liding/work/models/Baichuan2-7B-Chat'
query='解释一下“温故而知新”'

tokenizer = AutoTokenizer.from_pretrained(model_path, use_fast=False, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto", torch_dtype=torch.bfloat16, trust_remote_code=True)
model.generation_config = GenerationConfig.from_pretrained(model_path)

messages = []
messages.append({"role": "user", "content": query})

response = model.chat(tokenizer, messages)
print(response)

错误内容：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/liding/.cache/huggingface/modules/transformers_modules/Baichuan2-7B-Chat/modeling_baichuan.py", line 781, in chat
    outputs = self.generate(input_ids, generation_config=generation_config)
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 1648, in generate
    return self.sample(
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 2730, in sample
    outputs = self(
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/liding/.cache/huggingface/modules/transformers_modules/Baichuan2-7B-Chat/modeling_baichuan.py", line 684, in forward
    outputs = self.model(
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/liding/.cache/huggingface/modules/transformers_modules/Baichuan2-7B-Chat/modeling_baichuan.py", line 461, in forward
    layer_outputs = decoder_layer(
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/liding/.cache/huggingface/modules/transformers_modules/Baichuan2-7B-Chat/modeling_baichuan.py", line 273, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/liding/work/ds_venv/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/liding/.cache/huggingface/modules/transformers_modules/Baichuan2-7B-Chat/modeling_baichuan.py", line 233, in forward
    with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=True, enable_mem_efficient=True):
  File "/usr/local/lib/python3.10/contextlib.py", line 281, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/usr/local/lib/python3.10/contextlib.py", line 103, in __init__
    self.gen = func(*args, **kwds)
TypeError: sdp_kernel() got an unexpected keyword argument 'enable_mem_efficient'

[felixstander]

请问解决了吗，出现了同样的问题

[felixstander]

好像是torch版本的问题，2.0以上就没有问题了

[liding1992]

好像是torch版本的问题，2.0以上就没有问题了

是的，torch2.0.1可以。感谢

[mihara-bot]

pip install xformers可以解决这个问题

[code-gamer]

pip install xformers可以解决这个问题

安装xformers它会给你自动安装新版本的pytorch，好像是依赖，我安装时是这样的。python版本是3.8