You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
As a developer of an app that leverages LocalAI and Llama-2 for streaming completions, I want to give users the ability to "abort" or "cancel" the streaming response, so that my self-hosted instance is not spinning CPU / GPU cycles generating the rest of the stream that users won't even see.
importOpenAIfrom"openai";constcontent=`Please write JavaScript code that createsa scatter plot with D3.js.Use \`const\` and \`let\` instead of \`var\`.Use the arrow function syntax.## JavaScript code`;constopenai=newOpenAI({apiKey: "",baseURL: "http://192.168.0.140:8080/v1",});conststream=awaitopenai.chat.completions.create({model: "llama-2-7b-chat.ggmlv3.q4_0.bin",messages: [{role: "user", content }],stream: true,});letkeepGoing=truesetTimeout(()=>{// Approach A: This appears to do nothing.stream.controller.abort();// Approach B:// This stops the client from iterating,// but the server keeps computing the responsekeepGoing=false;},10*1000);forawait(constpartofstream){process.stdout.write(part.choices[0].delta.content);if(!keepGoing){break;}}
Additional context
The text was updated successfully, but these errors were encountered:
+1 on ability to cancel the stream, often we dont get what we want, and checking the LocalAI server its still processing previous generation burning a fair amount of resources in the process.
Is your feature request related to a problem? Please describe.
As a developer of an app that leverages LocalAI and Llama-2 for streaming completions, I want to give users the ability to "abort" or "cancel" the streaming response, so that my self-hosted instance is not spinning CPU / GPU cycles generating the rest of the stream that users won't even see.
Describe the solution you'd like
Ideally, I'd like to use the NodeJS OpenAI package API to abort the stream. As documented in https://github.com/openai/openai-node#streaming-responses , we should be able to invoke
or just
break;
out of the async loop.Describe alternatives you've considered
I've tried the following two approaches.
Additional context
The text was updated successfully, but these errors were encountered: