-
Notifications
You must be signed in to change notification settings - Fork 136
text generation webui_zh
text-generation-webui是一种基于Gradio的用于运行大模型的Web UI,支持Hugging Face,GGUF(llama.cpp)等常见格式。
text-generation-webui目前支持Windows/Linux/macOS/WSL系统。
1、下载并解压:https://github.com/oobabooga/text-generation-webui/archive/refs/heads/main.zip
2、根据不同操作系统,执行start_*.sh
脚本
3、根据提示选择GPU
What is your GPU?
A) NVIDIA
B) AMD (Linux/MacOS only. Requires ROCm SDK 5.6 on Linux)
C) Apple M Series
D) Intel Arc (IPEX)
N) None (I want to run models in CPU mode)
安装过程较长,请耐心等待。更详细的教程请参考:webui installation
text-generation-webui目前支持Hugging Face、GGUF等格式的模型。
将完整版Llama-3-Chinese-8B-Instruct放到text-generation-webui的models文件夹下,目录文件如下所示:
text-generation-webui
└── models
└── llama-3-chinese-8b-instruct
├── config.json
├── generation_config.json
├── model-00001-of-00004.safetensors
├── model-00002-of-00004.safetensors
├── model-00003-of-00004.safetensors
├── model-00004-of-00004.safetensors
├── model.safetensors.index.json
├── special_tokens_map.json
├── tokenizer_config.json
└── tokenizer.json
如果是GGUF格式,可作为一个单独文件直接放到对应目录中。相应权重可从完整模型下载 GGUF下载,然后重命名,文件目录如下:
text-generation-webui
└── models
└── llama-3-chinese-8b-instruct-q4_k.gguf
安装完毕之后,再次运行start_*.sh
,浏览器中访问日志显示的地址,即可进入网页界面。
15:59:59-435616 INFO Starting Text generation web UI
Running on local URL: http://127.0.0.1:7860
界面如下所示:
![image](https://private-user-images.githubusercontent.com/16095339/325136907-c8c7b7e7-9118-4e58-9c11-fe6a79261ba6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjUzNDU1NjUsIm5iZiI6MTcyNTM0NTI2NSwicGF0aCI6Ii8xNjA5NTMzOS8zMjUxMzY5MDctYzhjN2I3ZTctOTExOC00ZTU4LTljMTEtZmU2YTc5MjYxYmE2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTAzVDA2MzQyNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTliYmMzNDdhZjE4MGZiZmFmY2M0ZTg0NjNlNDk2ZjZkMGExYTI2NDdiYjI5Nzg4NGFjNmI2MGE3OTBhNTYyODEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.0Mp-3U9-K_ABM0hthYyHFOT53RaRuXXKRRdY0cqejjA)
接下来要进行一些配置:
1)选择model
选项卡,在下拉列表里选择llama-3-chinese-8b-instruct
,再点击load
按钮,即可加载模型。
2)选择parameters
选项卡,再选择instruction tempalte
选项卡,指令模板形如:
{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = '<|begin_of_text|>' + content %}{% endif %}{{ content }}{% endfor %}{{ '<|start_header_id|>assistant<|end_header_id|>
' }}
回到chat
选项卡,右侧的Mode
选择Instruct
,即可开始与模型对话。
更详细的用法请自行探索网页界面,或参考官方手册。如果遇到安装或者运行问题还请到原repo下搜索解决方案或提问。
- Model Reconstruction
- Model Quantization, Inference and Deployment
- System Performance
- Training Scripts
- FAQ