[BUG] TypeError: LlamaRotaryEmbedding.forward() got an unexpected keyword argument 'seq_len'

[timefliesfang]

Describe the bug
Encounter this when running genreration_speed.py. Input model is Llama-2-7b-chat-gptq. Complete error log:

Traceback (most recent call last):
File "AutoGPTQ/examples/benchmark/generation_speed.py", line 326, in
main()
File "AutoGPTQ/examples/benchmark/generation_speed.py", line 316, in main
benchmark_generation_speed(model, tokenizer, examples, generation_config)
File "AutoGPTQ/examples/benchmark/generation_speed.py", line 197, in benchmark_generation_speed
outputs_ids = model.generate(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 532, in generate
return self.model.generate(**kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1527, in generate
result = self._greedy_search(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/generation/utils.py", line 2411, in _greedy_search
outputs = self(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1196, in forward
outputs = self.model(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1016, in forward
layer_outputs = decoder_layer(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 739, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/auto_gptq/nn_modules/fused_llama_attn.py", line 76, in forward
cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
TypeError: LlamaRotaryEmbedding.forward() got an unexpected keyword argument 'seq_len'

Hardware details
Information about CPU and GPU, such as RAM, number, etc.
Nvidia V100

Software version
Version of relevant software such as operation system, cuda toolkit, python, auto-gptq, pytorch, transformers, accelerate, etc.

CUDA: 12.3 , Pytorch: 2.2.1, transformers: 4.39.1, accelerate: 0.28.0

[fxmarty]

Thank you for the report @timefliesfang, this is likely due to a breaking change in Transformers following https://github.com/huggingface/transformers/releases/tag/v4.38.0. I'll try to fix, but as a temporary workaround, can you try to use transformers==4.37.2?

[timefliesfang]

Yes, it works now. Thank you for your response:)

[WouterGlorieux]

@fxmarty Hi, I'm maintaining a template on runpod with text-generation-webui. A few hours ago the template broke again and I had to fix it by upgrading gradio and transformers.
I now have to use transformers 4.39.2 or text-generation-webui will not start anymore. Unfortunately this means I currently can not use the Llava-v1.5 anymore because of this issue.

Do you have any idea on when this issue will be fixed?