最新代码增量预训练qwen-7b后无法评测

[fengcai24]

报错如下：

output = old_forward(*args, **kwargs)

File "/root/.cache/huggingface/modules/transformers_modules/checkpoint-2800/modeling_qwen.py", line 381, in forward
context_layer = self.core_attention_flash(q, k, v)
File "/opt/conda/envs/llama_etuning/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/checkpoint-2800/modeling_qwen.py", line 95, in forward
assert all((i.dtype in [torch.float16, torch.bfloat16] for i in (q, k, v)))
AssertionError

[fengcai24]

模型训练采用bf16

[hiyouga]

在模型目录下的 config.json 中禁用 flashattn

[fengcai24]

已解决