RuntimeError: The size of tensor a (4096) must match the size of tensor b (4097) at non-singleton dimension 3

[edisonzf2020]

Reminder

• I have read the README and searched the existing issues.

Reproduction

python src/api_demo.py --model_name_or_path 01-ai/Yi-6B --adapter_name_or_path saves/yi-6b-agent/checkpoint-500 --template default --finetuning_type lora --quantization_bit 4
客户端推理报错：

File "/opt/works/AI/LLaMA-Factory/src/llmtuner/api/app.py", line 136, in chat_completion
   responses = chat_model.chat(
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
   return func(*args, **kwargs)
 File "/opt/works/AI/LLaMA-Factory/src/llmtuner/chat/chat_model.py", line 101, in chat
   generate_output = self.model.generate(**gen_kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/peft/peft_model.py", line 1148, in generate
   outputs = self.base_model.generate(*args, **kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
   return func(*args, **kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/transformers/generation/utils.py", line 1592, in generate
   return self.sample(
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/transformers/generation/utils.py", line 2696, in sample
   outputs = self(
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
   return self._call_impl(*args, **kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
   return forward_call(*args, **kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
   output = module._old_forward(*args, **kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1168, in forward
   outputs = self.model(
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
   return self._call_impl(*args, **kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
   return forward_call(*args, **kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/accelerate/hooks.py", line 166, in new_forward
   output = module._old_forward(*args, **kwargs)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 982, in forward
   causal_mask = self._update_causal_mask(attention_mask, inputs_embeds)
 File "/home/fanai/.conda/envs/llama_factory/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1075, in _update_causal_mask
   padding_mask = causal_mask[..., :mask_length].eq(0.0) * attention_mask[:, None, None, :].eq(0.0)
RuntimeError: The size of tensor a (4096) must match the size of tensor b (4097) at non-singleton dimension 3

Expected behavior

No response

System Info

No response

Others

No response

[ccccj]

我是在训练的时候遇到的这个报错，请问有进展了吗

[hiyouga]

该问题是由于数据的长度超过了模型的最大长度，可以通过修改模型的最大长度来解决
https://huggingface.co/01-ai/Yi-6B/blob/main/config.json#L11