int8量化版本finetuning报错：RuntimeError: self and mat2 must have the same dtype

[zlht812]

Traceback (most recent call last):
File "/data/ChatGLM-Tuning/finetune.py", line 117, in
main()
File "/data/ChatGLM-Tuning/finetune.py", line 110, in main
trainer.train()
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/transformers/trainer.py", line 1662, in train
return inner_training_loop(
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/transformers/trainer.py", line 1929, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/transformers/trainer.py", line 2699, in training_step
loss = self.compute_loss(model, inputs)
File "/data/ChatGLM-Tuning/finetune.py", line 54, in compute_loss
return model(
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/peft/peft_model.py", line 678, in forward
return self.base_model(
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/chatglm-6b-int8/modeling_chatglm.py", line 1190, in forward
transformer_outputs = self.transformer(
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/chatglm-6b-int8/modeling_chatglm.py", line 985, in forward
layer_ret = torch.utils.checkpoint.checkpoint(
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 249, in checkpoint
return CheckpointFunction.apply(function, preserve, *args)
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 107, in forward
outputs = run_function(*args)
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/chatglm-6b-int8/modeling_chatglm.py", line 627, in forward
attention_outputs = self.attention(
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/.cache/huggingface/modules/transformers_modules/chatglm-6b-int8/modeling_chatglm.py", line 445, in forward
mixed_raw_layer = self.query_key_value(hidden_states)
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/aigpu310/lib/python3.10/site-packages/peft/tuners/lora.py", line 565, in forward
result = F.linear(x, transpose(self.weight, self.fan_in_fan_out), bias=self.bias)
RuntimeError: self and mat2 must have the same dtype
请教～

[calvinzhan]

这是因为量化版会把query_key_value改掉，封装；而lora又会改回来。这样input是float16, weight是int8，不能运算。不知有人成功解救这问题的吗？

[zlht812]

换了另外的训练脚本和f16权重，可以了。原版的训练脚本不支持int 8

[calvinzhan]

@zlht812
换了训练版本可以支持int8量化版模型了？换了f16权重，这里能说下是怎么做的吗？方便加下vx，交流下？我的是229402265

[songyi1999]

我也想问知道这个问题的答案,能发下吗?

[zlht812]

用的这个支持int8的lora：https://github.com/ssbuild/chatglm_finetuning
目前的情况是，lora训练完成，推理时，预训练模式使用f16，但使用int8方式加载。lora使用half()成功加载，但推理时，又报同样的错误。

推理用的预训练模型和训练lora用的一样,怀疑是lora()本身就是int8，所以去掉half(),结果显卡挂了：CUDA out of memory
等新服务器上线后，再测试下。

[Xzaohui]

有什么解决方案吗