output 生成内容为乱码 #2

ZenXir · 2023-03-23T11:54:25Z

大佬老师你好：

input:
中国的首都是哪儿？

output:
��

正常运行起来后，通过chrome浏览器访问：http://127.0.0.1:7860
生成结果为乱码

改用360浏览器，结果均为乱码，
需要设置什么吗？

Facico · 2023-03-23T12:01:34Z

这个是在linux环境下训练的吗？然后方便把运行命令，还有webui上的参数提供一下吗

ZenXir · 2023-03-23T12:18:20Z

运行环境是win10下的 wsl 环境：具体配置为：附件里有个环境配置说明.txt 我把环境的配置流程都记录下来了大佬老师 -------------------------------------------------------------------- 可能会出现这样的错误是依赖库自身问题在它的issus里有解决方案： 1、AttributeError: /root/anaconda3/envs/Chinese-Vicuna/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

…

------$ 解决方案在：bitsandbytes-foundation/bitsandbytes#156 2、AttributeError: module 'gradio' has no attribute 'components' ------$ pip install gradio==3.23.0 安装最新版本 -------------------------------------------------------------------- 运行命令是：bash generate.sh 运行参数用附件截图了老师：output乱码和参数截图.png-------------------------------------------------------------------- 基础模型是：git clone https://huggingface.co/decapoda-research/llama-7b-hf LORA模型是：当前目录的 ./lora-Vicuna/checkpoint-4000 generate.sh 脚本内容是（也提交附件了大佬老师）： BASE_MODEL="/mnt/e/zllama-models/llama-7b-hf"LORA_PATH="./lora-Vicuna/checkpoint-4000" cp ./config-sample/adapter_config.json $LORA_PATH CUDA_VISIBLE_DEVICES=0 python generate.py \     --model_path $BASE_MODEL \     --lora_path $LORA_PATH Original From:"Facico"< ***@***.*** >; Date:2023/3/23 20:01 To:"Facico/Chinese-Vicuna"< ***@***.*** >; CC:"SandMist"< ***@***.*** >;"Author"< ***@***.*** >; Subject:Re: [Facico/Chinese-Vicuna] output 生成内容为乱码 (Issue #2) 这个是在linux环境下训练的吗？然后方便把运行命令，还有webui上的参数提供一下吗 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

Facico · 2023-03-23T13:01:41Z

@ZenXir
非常感谢你提供了配置问题即解决方法。不过我这边好像没有看到你的附件，不过我感觉问题应该不在参数上。目前怀疑的就是环境上的问题。
我这边相关库的环境是transformers==4.28.0, tokenizers==0.13.2, sentencepiece==0.1.97，如果这些库对齐的没有效果，你可以提供更详细的库的信息给我。
还可能存在的原因是中文编码问题，你可以试试用英文提问或让他英文回答能否成功，可以在generate.py的evaluate函数中将output在终端输出，看看有没有可能是webui显示的问题。

ZenXir · 2023-03-24T03:22:18Z

大佬老师，我刚按你说的方式，在命令行和网页输出的是乱码，英文问答也是，
这个是命令行打印的：

Instruction: who are you
Response: ��

pip环境参数是：

pip list | grep transformers

transformers 4.28.0.dev0

pip list | grep tokenizers

tokenizers 0.13.2

pip list | grep sentencepiece

sentencepiece 0.1.97

系统环境是win11下的wsl，wsl里的环境创建过程是这样的：

在win11环境使用wsl：http://events.jianshu.io/p/6479d450da2a

进入 wsl，使用root

下载

wget Anaconda3-2022.05-Linux-x86_64.sh 可以找最新的下载

运行

bash Anaconda3-2022.05-Linux-x86_64.sh

提示输入yes同意条款，输入yes同意安装路径。
UBUNTU安装ANACONDA后终端出现BASE
屏蔽>>> conda initialize >>>到

<<< conda initialize <<<

vim .bashrc
export PATH="/home/test/anaconda3/bin:$PATH"
source .bashrc
关闭ubuntu，重新打开ubuntu就可以用了

ubuntu安装gcc

sudo apt-get update
sudo apt-get install gcc
sudo apt-get install g++

升级系统内核

sudo apt-get dist-upgrade

如果有安装CUDA 卸载老版本CUDA: 【https://blog.csdn.net/xian0710830114/article/details/124094603】
sudo apt-get --purge remove "cublas" "cuda*"

安装整个CUDA包，在下面这个链接选择对应的平台和系统，会给出安装命令：
CUDA Toolkit 11.8 Downloads:
https://developer.nvidia.com/cuda-downloads

Installation Instructions:
--$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
--$ sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
--$ wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-ubuntu2004-11-8-local_11.8.0-520.61.05-1_amd64.deb
--$ sudo dpkg -i cuda-repo-ubuntu2004-11-8-local_11.8.0-520.61.05-1_amd64.deb
--$ sudo cp /var/cuda-repo-ubuntu2004-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
--$ sudo apt-get update
--$ sudo apt-get -y install cuda

安装pytorch:这个是对应当前机器显卡 4090 ，从官网 https://pytorch.org/
得出的命令 : pytorch-cuda=11.8 这里的版本号需要与上面系统的 CUDA Toolkit 版本号一致
--$ conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch-nightly -c nvidia
--$ conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia 【这个项目使用的是cu117】

对不同的conda python环境，在 D:\whl 目录下，安装对应的轮子
轮子官网地址：https://download.pytorch.org/whl/torch_stable.html

conda create -n Chinese-Vicuna python=3.9
conda activate Chinese-Vicuna

#如果你已经用pip安装过不少软件，并且没有选择不缓存的话，可以用下面的命令来删除缓存
#Linux and Unix
rm -rf ~/.cache/pip

#Mac Os
rm -rf ~/Library/Caches/pip

#Windows
rd/s/q %LocalAppData%/pipCache

配置环境：

cd Chinese-Vicuna

pip install datasets
pip install loralib
pip install sentencepiece

安装transformers，直接从主干拉取 transformers
git clone https://github.com/huggingface/transformers.git
cd transformers
python setup.py install

pip install accelerate
pip install -U bitsandbytes

安装peft，直接从主干拉取 peft
git clone https://github.com/huggingface/peft.git
cd peft
python setup.py install

pip install gradio
pip install appdirs
pip install chardet
pip install cchardet
pip install --user pillow

#注意事项，解决运行问题：

1、AttributeError: /root/anaconda3/envs/Chinese-Vicuna/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cget_col_row_stats

先确认用的cuda是哪个版本：

import torch
print(torch.cuda.is_available())

------$ python /mnt/e/pytorch_test.py
True
2.0.0+cu117

------$ 问题解释在：bitsandbytes-foundation/bitsandbytes#156
------$ locate libbitsandbytes_cuda117.so
/root/anaconda3/envs/Chinese-Vicuna/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
------$ locate bitsandbytes_cpu.so
/root/anaconda3/envs/Chinese-Vicuna/lib/python3.9/site-packages/bitsandbytes/libbitsandbytes_cpu.so
------$ cd /root/anaconda3/envs/Chinese-Vicuna/lib/python3.9/site-packages/bitsandbytes/
------$ mv libbitsandbytes_cpu.so libbitsandbytes_cpu.so.bak
------$ cp libbitsandbytes_cuda117.so libbitsandbytes_cpu.so

2、AttributeError: module 'gradio' has no attribute 'components'
------$ pip install gradio==3.23.0 安装最新版本

Facico · 2023-03-24T03:54:12Z

@ZenXir
这是一份简单的调用llama和lora代码，可以测试一下有没有问题

import sys
import torch
from peft import PeftModel
import transformers
from transformers import LlamaTokenizer, LlamaForCausalLM

tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
BASE_MODEL = "decapoda-research/llama-7b-hf"

model = LlamaForCausalLM.from_pretrained(
    BASE_MODEL,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
)
model.eval()
inputs = "Hello, Where is the capital of the United States?" #"你好,美国的首都在哪里？"
input_ids = tokenizer(inputs, return_tensors="pt")['input_ids']
print(input_ids)
generation_output = model.generate(
            input_ids=input_ids,
            max_new_tokens=15,
        )
print(generation_output)
print(tokenizer.decode(generation_output[0]))

model = PeftModel.from_pretrained(
        model,
        "./lora-alpaca/checkpoint-4000",
        torch_dtype=torch.float16,
        device_map={'': 0}
    )

inputs = "你好,中国的首都在哪里？" #"你好,美国的首都在哪里？"
input_ids = tokenizer(inputs, return_tensors="pt")['input_ids']
print(input_ids)
generation_output = model.generate(
            input_ids=input_ids,
            max_new_tokens=15,
        )
print(generation_output)
print(tokenizer.decode(generation_output[0]))

正常输出的话会输出下面内容

tensor([[    1, 15043, 29892,  6804,   338,   278,  7483,   310,   278,  3303,
          3900, 29973,    13,  1576,  7483,   310,   278,  3303,  3900,   338,
          7660, 29892,   360, 29889, 29907, 29889,    13]])
 Hello, Where is the capital of the United States?
The capital of the United States is Washington, D.C.

tensor([[    1, 29871, 30919, 31076, 29892, 30275, 30356, 30210, 31688, 30769,
         30505,   232,   150,   173, 30755, 30882]])
tensor([[    1, 29871, 30919, 31076, 29892, 30275, 30356, 30210, 31688, 30769,
         30505,   232,   150,   173, 30755, 30882,    13, 30919, 31076, 30214,
         30275, 30356, 30210, 31688, 30769, 30392, 30662, 30675, 30267,    13,
         30919]])
 你好,中国的首都在哪里？
你好，中国的首都是北京。

使用CUDA_VISIBLE_DEVICES=0 python simple_test.py来跑

ZenXir · 2023-03-24T05:02:59Z

/root/anaconda3/envs/Chinese-alpaca-lora/lib/python3.9/site-packages/transformers-4.28.0.dev0-py3.9.egg/transformers/generation/utils.py:1211: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
warnings.warn(
/root/anaconda3/envs/Chinese-alpaca-lora/lib/python3.9/site-packages/transformers-4.28.0.dev0-py3.9.egg/transformers/generation/utils.py:1384: UserWarning: You are calling .generate() with the input_ids being on a device type different than your model's device. input_ids is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. Please make sure that you have put input_ids to the correct device by calling for example input_ids = input_ids.to('cuda') before running .generate().
warnings.warn(
tensor([[ 1, 15043, 29892, 6804, 338, 278, 7483, 310, 278, 3303,
3900, 29973, 13, 1576, 7483, 310, 278, 3303, 3900, 338,
7660, 29892, 360, 29889, 29907, 29889, 13]])
Hello, Where is the capital of the United States?
The capital of the United States is Washington, D.C.

Traceback (most recent call last):
File "/root/anaconda3/envs/Chinese-alpaca-lora/lib/python3.9/site-packages/peft-0.3.0.dev0-py3.9.egg/peft/utils/config.py", line 99, in from_pretrained
File "/root/anaconda3/envs/Chinese-alpaca-lora/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 112, in _inner_fn
validate_repo_id(arg_value)
File "/root/anaconda3/envs/Chinese-alpaca-lora/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 160, in validate_repo_id
raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': './lora-alpaca/checkpoint-8000'. Use repo_type argument if needed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/mnt/e/Chinese-Vicuna/eval_test.py", line 27, in
model = PeftModel.from_pretrained(
File "/root/anaconda3/envs/Chinese-alpaca-lora/lib/python3.9/site-packages/peft-0.3.0.dev0-py3.9.egg/peft/peft_model.py", line 135, in from_pretrained
File "/root/anaconda3/envs/Chinese-alpaca-lora/lib/python3.9/site-packages/peft-0.3.0.dev0-py3.9.egg/peft/utils/config.py", line 101, in from_pretrained
ValueError: Can't find config.json at './lora-alpaca/checkpoint-8000'

上的面例子，英文问答正确，中文问答时，崩溃了大佬老师 checkpoint-xxx 里其实有 adapter_config.json 我正在找原因
csdn有人说，用完整路径，不能用 ./ 相对路径试了下不起作用我正在google

代码是（只修改了model的路径大佬老师）：

import sys
import torch
from peft import PeftModel
import transformers
from transformers import LlamaTokenizer, LlamaForCausalLM

tokenizer = LlamaTokenizer.from_pretrained("/mnt/e/zllama-models/llama-7b-hf")
BASE_MODEL = "/mnt/e/zllama-models/llama-7b-hf"

model = LlamaForCausalLM.from_pretrained(
    BASE_MODEL,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
)
model.eval()
inputs = "Hello, Where is the capital of the United States?" #"你好,美国的首都在哪里？"
input_ids = tokenizer(inputs, return_tensors="pt")['input_ids']
print(input_ids)
generation_output = model.generate(
            input_ids=input_ids,
            max_new_tokens=15,
        )
print(generation_output)
print(tokenizer.decode(generation_output[0]))

model = PeftModel.from_pretrained(
        model,
        "/mnt/e/Chinese-Vicuna/lora-Vicuna/lora-alpaca/checkpoint-8000",
        torch_dtype=torch.float16,
        device_map={'': 0}
    )

inputs = "你好,中国的首都在哪里？" #"你好,美国的首都在哪里？"
input_ids = tokenizer(inputs, return_tensors="pt")['input_ids']
print(input_ids)
generation_output = model.generate(
            input_ids=input_ids,
            max_new_tokens=15,
        )
print(generation_output)
print(tokenizer.decode(generation_output[0]))

ZenXir · 2023-03-24T05:52:35Z

可以正常运行大佬老师

"/mnt/e/Chinese-Vicuna/lora-Vicuna/lora-alpaca/checkpoint-8000",
这个路径我指定时，出现了两个 lora-Vicuna 重复了

改成正确的：
lora-alpaca/checkpoint-8000 【前面不加 ./ 表现相对路径】
或
"/mnt/e/Chinese-Vicuna/lora-Vicuna/checkpoint-8000", 【使用绝对路径】

我再检查下web的显示，感谢大佬老师
不过运行的结果应该还需要再训练：
这两个问题的结果有时是这样的：

Hello, Where is the capital of the United States?
The capital of the United States is Washington, D.C.

中国的首都是哪里？
中国

Facico · 2023-03-24T05:58:04Z

OK，这个其实是一个比较简单的例子，生成的时候没有加参数控制，在webui那里会有参数控制的

ZenXir · 2023-03-24T05:59:55Z

明白了感谢大佬老师 Original From:"Facico"< ***@***.*** >; Date:2023/3/24 13:58 To:"Facico/Chinese-Vicuna"< ***@***.*** >; CC:"SandMist"< ***@***.*** >;"State change"< ***@***.*** >; Subject:Re: [Facico/Chinese-Vicuna] output 生成内容为乱码 (Issue #2) OK，这个其实是一个比较简单的例子，生成的时候没有加参数控制，在webui那里会有参数控制的 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: ***@***.***>

chenzk1993 · 2023-04-07T08:20:46Z

tensor([[ 1, 15043, 29892, 6804, 338, 278, 7483, 310, 278, 3303,
3900, 29973, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0]])
Hello, Where is the capital of the United States? ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇

chenzk1993 · 2023-04-07T08:21:03Z

tensor([[ 1, 29871, 30919, 31076, 29892, 30275, 30356, 30210, 31688, 30769,
30505, 232, 150, 173, 30755, 30882]])
/data/miniconda3/envs/OpenChatKit/lib/python3.10/site-packages/transformers/generation/utils.py:1384: UserWarning: You are calling .generate() with the input_ids being on a device type different than your model's device. input_ids is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. Please make sure that you have put input_ids to the correct device by calling for example input_ids = input_ids.to('cuda') before running .generate().
warnings.warn(
tensor([[ 1, 29871, 30919, 31076, 29892, 30275, 30356, 30210, 31688, 30769,
30505, 232, 150, 173, 30755, 30882, 13, 13, 2277, 29937,
2799, 4080, 29901, 13, 13, 30919, 31383, 30698, 30742, 234,
176]])
你好,中国的首都在哪里？

chenzk1993 · 2023-04-07T08:23:06Z

@Facico 运行simple_test.py 使用7B模型返回正常，但13B返回如下，是不是说明13B文件下载有问题?
tensor([[ 1, 15043, 29892, 6804, 338, 278, 7483, 310, 278, 3303,
3900, 29973, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0]])
Hello, Where is the capital of the United States? ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇

tensor([[ 1, 29871, 30919, 31076, 29892, 30275, 30356, 30210, 31688, 30769,
30505, 232, 150, 173, 30755, 30882]])
/data/miniconda3/envs/OpenChatKit/lib/python3.10/site-packages/transformers/generation/utils.py:1384: UserWarning: You are calling .generate() with the input_ids being on a device type different than your model's device. input_ids is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. Please make sure that you have put input_ids to the correct device by calling for example input_ids = input_ids.to('cuda') before running .generate().
warnings.warn(
tensor([[ 1, 29871, 30919, 31076, 29892, 30275, 30356, 30210, 31688, 30769,
30505, 232, 150, 173, 30755, 30882, 13, 13, 2277, 29937,
2799, 4080, 29901, 13, 13, 30919, 31383, 30698, 30742, 234,
176]])
你好,中国的首都在哪里？

Instruction:

你需要回��

Facico · 2023-04-07T08:26:50Z

相似的问题，可以参考issue1 issue2

ZenXir closed this as completed Mar 24, 2023

Facico mentioned this issue Apr 3, 2023

基于13B的LLAMA模型fine-tune，loss特别大，而lr初始就是0，这是正常的吗？ #32

Closed

Facico mentioned this issue Apr 18, 2023

关于generate生成的结果的问题 #71

Closed

Facico added the good first issue Good for newcomers label Apr 21, 2023

18065013 mentioned this issue Jul 17, 2023

多卡finetune_chat时报mat1 and mat2 shapes cannot be multiplied (1024x2 and 1x11008) #240

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

output 生成内容为乱码 #2

output 生成内容为乱码 #2

ZenXir commented Mar 23, 2023

Facico commented Mar 23, 2023

ZenXir commented Mar 23, 2023 via email

Facico commented Mar 23, 2023

ZenXir commented Mar 24, 2023

Facico commented Mar 24, 2023 •

edited

Loading

ZenXir commented Mar 24, 2023

ZenXir commented Mar 24, 2023

Facico commented Mar 24, 2023

ZenXir commented Mar 24, 2023 via email

chenzk1993 commented Apr 7, 2023

chenzk1993 commented Apr 7, 2023

chenzk1993 commented Apr 7, 2023

Facico commented Apr 7, 2023

output 生成内容为乱码 #2

output 生成内容为乱码 #2

Comments

ZenXir commented Mar 23, 2023

Facico commented Mar 23, 2023

ZenXir commented Mar 23, 2023 via email

Facico commented Mar 23, 2023

ZenXir commented Mar 24, 2023

pip list | grep transformers

pip list | grep tokenizers

pip list | grep sentencepiece

在win11环境 使用wsl：http://events.jianshu.io/p/6479d450da2a

下载

运行

<<< conda initialize <<<

ubuntu安装gcc

升级系统内核

配置环境：

#注意事项，解决运行问题：

Facico commented Mar 24, 2023 • edited Loading

ZenXir commented Mar 24, 2023

上的面例子，英文问答正确，中文问答时，崩溃了大佬老师 checkpoint-xxx 里其实有 adapter_config.json 我正在找原因 csdn有人说，用完整路径，不能用 ./ 相对路径 试了下不起作用 我正在google

ZenXir commented Mar 24, 2023

可以正常运行大佬老师

Facico commented Mar 24, 2023

ZenXir commented Mar 24, 2023 via email

chenzk1993 commented Apr 7, 2023

chenzk1993 commented Apr 7, 2023

chenzk1993 commented Apr 7, 2023

Instruction:

Facico commented Apr 7, 2023

在win11环境使用wsl：http://events.jianshu.io/p/6479d450da2a

Facico commented Mar 24, 2023 •

edited

Loading

上的面例子，英文问答正确，中文问答时，崩溃了大佬老师 checkpoint-xxx 里其实有 adapter_config.json 我正在找原因
csdn有人说，用完整路径，不能用 ./ 相对路径试了下不起作用我正在google