Add new providers: Koboldcpp and Goinfer #22
Conversation
| @@ -231,3 +232,12 @@ export const randomString = (): string => { | |||
| } | |||
| return result; | |||
| }; | |||
|
|
|||
| export function llamaMaxTokens(prompt: string, ctx: number) { | |||
There was a problem hiding this comment.
I like this as a temporary solution but as mentioned here (#21 (comment)), I'm not sure this will work long term.
Not an issue for this PR.
| return ctx - n; | ||
| } | ||
|
|
||
| export function formatPrompt(prompt: string, template: string, systemMessage: string) { |
There was a problem hiding this comment.
This will likely change or be removed entirely if/when the solution outlined in #21 (comment) is implemented, but works great for the purposes of this PR.
| this.apiUrl = options?.apiUrl ?? DEFAULT_API_URL; | ||
| } | ||
|
|
||
| /* async complete(params: KoboldInferParams, options?: { signal?: AbortSignal }): Promise<string> { |
There was a problem hiding this comment.
I'm fine with leaving this commented code here as a point of reference. It has value in that sense.
Curious: do you see a reason for wingman to ever support anything other than a streamed response? Do you think each provider should support buffered responses like this, or no?
There was a problem hiding this comment.
I see no reason to use anything else than streamed response: every backend provider supports it, this is a de facto standard. I am in favor of completely remove the completion method as it has no value actually and it would simplify the code, which is better for maintenance and implementation of new providers
No description provided.