加载8bit量化/离线量化模型报错：RuntimeError: probability tensor contains either inf, nan or element < 0

[ywancit]

背景：可以正常使用无量化模型。保存8bit量化模型过程无报错。按照官方文档中加载8bit量化/离线量化模型报错：RuntimeError: probability tensor contains either inf, nan or element < 0

代码如下：
model = AutoModelForCausalLM.from_pretrained(r".\Baichuan2-13B-Chat",
load_in_8bit=True, device_map="auto", trust_remote_code=True)
model.save_pretrained(r'.\8bit')
model = AutoModelForCausalLM.from_pretrained(r'.\8bit', device_map="auto", trust_remote_code=True)
model.generation_config = GenerationConfig.from_pretrained(r".\Baichuan2-13B-Chat")
tokenizer = AutoTokenizer.from_pretrained(r".\Baichuan2-13B-Chat",
use_fast=False, trust_remote_code=True)

messages = []
messages.append({"role": "user", "content": "解释一下“温故而知新”"})
response = model.chat(tokenizer, messages)
print(response)

报错如下：
in GenerationMixin.sample(self, input_ids, logits_processor, stopping_criteria, logits_warper, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, streamer, **model_kwargs)
2676 # sample
2677 probs = nn.functional.softmax(next_token_scores, dim=-1)
-> 2678 next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
2680 # finished sentences should have their next token be a padding token
2681 if eos_token_id is not None:

RuntimeError: probability tensor contains either inf, nan or element < 0

[bihui9968]

我也遇到了

[jameswu2014]

是最新的代码吗？

[ywancit]

是最新的代码吗？

是的，4小时之前刚clone的模型文件和repo。

[jameswu2014]

我这边用你发的代码跑了一下，是正常的。

[jameswu2014]

你检查一下是不是哪些改了？

[ldisgod]

解决了么？

[Dori-Nilou]

请问解决了吗

[ldisgod]

没有

…

---- 回复的原邮件 ---- | 发件人 | ***@***.***> | | 发送日期 | 2023年10月11日 19:30 | | 收件人 | baichuan-inc/Baichuan2 ***@***.***> | | 抄送人 | ldisgod ***@***.***>, Comment ***@***.***> | | 主题 | Re: [baichuan-inc/Baichuan2] 加载8bit量化/离线量化模型报错：RuntimeError: probability tensor contains either `inf`, `nan` or element < 0 (Issue #48) | 请问解决了吗 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>