按照官网的方式搞的，但是答案无法输出。 #143
Labels
question
Further information is requested
Comments
|
transformers 版本 4.40.2 |
|
您好，我最近写一个单个文件的gradio给您，预计明后天 |
|
您好，下面是取消流式输出的代码： import gradio as gr
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
# @torch.no_grad()
def generate_response(instruction, text="", temperature=1.0, top_p=0.9, top_k=50, num_beams=1, max_new_tokens=50, repetition_penalty=1.0):
with torch.no_grad():
if text != "":
input_text = f"Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Input:\n{text}\n\n### Response:\n"
else:
input_text = f"Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{instruction}\n\n### Response:\n"
input_ids = tokenizer.encode(input_text, return_tensors='pt').to('cuda')
output_ids = model.generate(
input_ids,
max_length=input_ids.shape[1] + max_new_tokens,
temperature=temperature,
top_k=top_k,
top_p=top_p,
num_beams=num_beams,
repetition_penalty=repetition_penalty,
)
output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
return output_text[len(input_text):]
if __name__ == '__main__':
model_name = "zjunlp/knowlm-13b-zhixi"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name, torch_dtype=torch.bfloat16, device_map="auto",
load_in_8bit=True
)
interface = gr.Interface(
fn=generate_response,
inputs=[
gr.Textbox(label="Instruction", placeholder="Enter instruction here...", lines=2, value="""从给定的文本中提取出可能的实体和实体类型，可选的实体类型为['地点', '人名']，以（实体，实体类型）的格式回答。"""),
gr.Textbox(label="Optional Text", placeholder="Enter optional text here...", lines=2, optional=True, value="""John昨天在纽约的咖啡馆见到了他的朋友Merry。他们一起喝咖啡聊天，计划着下周去加利福尼亚（California）旅行。他们决定一起租车并预订酒店。他们先计划在下周一去圣弗朗西斯科参观旧金山大桥，下周三去洛杉矶拜访Merry的父亲威廉。"""),
gr.Slider(label="Temperature", minimum=0.1, maximum=2.0, value=1.0, step=0.1),
gr.Slider(label="Top p", minimum=0.0, maximum=1.0, value=0.9, step=0.01),
gr.Slider(label="Top k", minimum=0, maximum=100, value=50, step=1),
gr.Slider(label="Number of Beams", minimum=1, maximum=10, value=1),
gr.Slider(label="Max New Tokens", minimum=1, maximum=512, value=50),
gr.Slider(label="Repetition Penalty", minimum=0.1, maximum=1.6, value=1.0, step=0.1)
],
outputs="text",
title="Zhixi",
description="<center>https://github.com/zjunlp/knowlm</center>"
)
interface.launch() |
|
大佬还是不行，也不知道是环境问题还是啥问题： 如果用knowlm-13b-zhixi 模型 但是如果用generate_lora.py ,虽然有些浸膏，但是 就能正常推理出来。 |
|
如果使用oneke，请执行下面的命令： CUDA_VISIBLE_DEVICES=0 python examples/generate_lora_web.py --base_model zjunlp/oneke --model_tag oneke如果使用zhixi，请执行下面的命令： CUDA_VISIBLE_DEVICES=0 python examples/generate_lora_web.py --base_model zjunlp/knowlm-13b-zhixi --model_tag zhixi |
|
请问您的问题解决了吗 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The text was updated successfully, but these errors were encountered: