Local LLM models?, integration with Oobabooga if it's required

[juangea]

Describe the feature you'd like
To be able to use all this system locally, so we can use local models like Wizard-Vicuna and not having to share our data with OpenAI or other sites or clouds.

Maybe an option to avoid having to do a full local LLM implementation is to make it communicate with Oobabooga with it's API, not sure thought, but I suspect it's similar to talking with ChatGPT.

Will this be implemented at some point?

Thanks!

[HenryHengZJ]

We recently just merged the integration using https://github.com/go-skynet/LocalAI

#123

[juangea]

Oh, awesome, any guide on how to use it? or just install LocalAI and then there is a node to connect to it?

Or maybe it's just part of Flowise? :)

Thanks!

[HenryHengZJ]

You'll need to have a Docker container running LocalAI. I'd suggest follow the quick guide here - https://github.com/go-skynet/LocalAI#example-use-gpt4all-j-model

Once you have that running, you can add the LocalAI node on Flowise, and fill in the url and model name. You can refer to the video attached in the PR - #123

[juangea]

Thanks :)

[Egalitaristen]

Hey @HenryHengZJ

So I've tried to replace the ChatOpenAI with the ChatLocalAI chat model in the AutoGPT template but it didn't work since I can't connect the output from the local wrapper to the Chat Model input in the AutoGPT wrapper.

I gave ChatGPT the code (since I'm not a dev) for both wrappers (Local and OpenAI) and asked it to compare them, here's what it said.

Upon a quick review of your code, it seems there might be a mistake in how you're initializing your model in the ChatLocalAI_ChatModels class.

In the ChatOpenAI_ChatModels class, the model instance is initialized with an object that has a type of OpenAIChatInput, which matches the model type ChatOpenAI.

However, in the ChatLocalAI_ChatModels class, you're initializing an OpenAIChat model, but you're still passing it an OpenAIChatInput type object.

Considering that you're using a different model (OpenAIChat), it's likely that the input parameters required are different from those required by the ChatOpenAI model, hence the confusion.

Here is the problematic code:

javascript
Copy code
const obj: Partial & { openAIApiKey?: string } = {
temperature: parseInt(temperature, 10),
modelName,
openAIApiKey: 'sk-'
}

const model = new OpenAIChat(obj, { basePath })
It seems like you've used the same object structure as the OpenAIChatInput in the ChatOpenAI_ChatModels class, while the model you're initializing expects a different structure.

You may need to adjust this to suit the OpenAIChat model's requirements. If you're unsure what these are, it would be worth checking the documentation for the OpenAIChat model or even its source code if it's available.

It's also possible that the OpenAIChat doesn't expect or require the openAIApiKey property, given that this model is for local LLMs, which probably don't require an API key for OpenAI.

You also need to check if the imported OpenAIChat from 'langchain/llms/openai' is the correct class. The second wrapper seems to be for a local model, yet it's importing a class that seems to be related to OpenAI's models.

Please verify these assumptions by checking the OpenAIChat and OpenAIChatInput class definitions, to ensure they match the expected input properties and that the correct class is being instantiated.

[HenryHengZJ]

@MightEnlightenYou thanks for the bug! Its now fixed as #141 merged

[Egalitaristen]

@HenryHengZJ Thanks for fixing it!