Create extra container for localAI and integrate via openai app

[szaimen]

See https://github.com/go-skynet/LocalAI/blob/master/docker-compose.yaml
https://quay.io/repository/go-skynet/local-ai?tab=tags, search for v1.19.
https://github.com/nextcloud/integration_openai

[julien-nc]

• Whisper: According to https://localai.io/model-compatibility/index.html#model-compatibility-table they are using https://github.com/ggerganov/whisper.cpp so I assume you can try using the same models as used in https://github.com/nextcloud/stt_whisper (which are the ones provided by the Whisper.cpp project)
• Stable Diffusion: LocalAI uses https://github.com/mudler/go-stable-diffusion
• GPT-like Text completion: there's a lot of compatible models. I successfully tried using the model they mention in the README: https://github.com/go-skynet/LocalAI#example-use-gpt4all-j-model

[szaimen]

Thanks a lot @julien-nc! :)

Do you know if we need to compile https://github.com/mudler/go-stable-diffusion ourselves or if they ship binaries somewhere?

The others fortunately seem to have binaries already :)

[julien-nc]

I have no idea but I would assume they have it compiled in their docker image already. go-stable-diffusion uses a very permissive BSD license.

Do you know for a fact that it's not already in the image?

[szaimen]

I have no idea but I would assume they have it compiled in their docker image already. go-stable-diffusion uses a very permissive BSD license.

That would be cool of course :)

Do you know for a fact that it's not already in the image?

I simply have not checked yet :D

[Zoey2936]

I would say that an integration of it would be easy, but if I compare the generated results with the cpu usage, an implementation is currently worthless until the models are better trained. I talk about GPT, Stable Diffusion works great if I use vladmandics fork of automatic1111 with my Nvidia GPU, not sure about the CPU-based localai implementation of it. But what I can say, local hosted AI can only be used toghter with a GPU (it takes too long with a CPU) and only image generative AI can create useful results, but only with good checkpoints.

[DenuxPlays]

I would say that an integration of it would be easy, but if I compare the generated results with the cpu usage, an implementation is currently worthless until the models are better trained. I talk about GPT, Stable Diffusion works great if I use vladmandics fork of automatic1111 with my Nvidia GPU, not sure about the CPU-based localai implementation of it. But what I can say, local hosted AI can only be used toghter with a GPU (it takes too long with a CPU) and only image generative AI can create useful results, but only with good checkpoints.

Can confirm!
I recently tested localAI on my server (no gpu, 32GB Ram, Intel D-1521)
I know not the best CPU but way enough to run AIO.
I only tested the GPT models but I took a very long time to generate even small answers...
I haven't tested it with my GPU on my PC but with only a CPU it's barely usable.

[szaimen]

I recently tested localAI on my server (no gpu, 32GB Ram, Intel D-1521)
I know not the best CPU but way enough to run AIO.
I only tested the GPT models but I took a very long time to generate even small answers...
I haven't tested it with my GPU on my PC but with only a CPU it's barely usable.

Thanks for this test! Since I dont expect most systems that run AIO to actually have a decent GPU, I will postpone this feature for now since it does not seem to make sense on most systems.

[FlattusBlastus]

Those on Windows Docker Desktop cant pass the GPU at all. CPU is the only route available.
What port are you thinking of running on? I am looking at trying https://localai.io/howtos/easy-setup-docker-cpu/

[szaimen]

Hi, I just wanted to mention that we have this now: https://github.com/nextcloud/all-in-one/tree/main/community-containers#how-to-add-containers. So the community could potentially add localAI as additional container. Feel free to ping me if you should need help on this!

[szaimen]

I've created #3495 now.

[szaimen]

This is now released with v7.5.0 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel

[FlattusBlastus]

Going this route first due to no Linux support. https://ryzenai.docs.amd.com/en/latest/inst.html

…

________________________________ From: Simon L. ***@***.***> Sent: Thursday, September 28, 2023 5:16:05 AM To: nextcloud/all-in-one ***@***.***> Cc: FlattusBlastus ***@***.***>; Comment ***@***.***> Subject: Re: [nextcloud/all-in-one] Create extra container for localAI and integrate via openai app (Discussion #2805) Hi, I just wanted to mention that we have this now: https://github.com/nextcloud/all-in-one/tree/main/community-containers#how-to-add-containers. So the community could potentially add localAI as additional container. Feel free to ping me if you should need help on this! — Reply to this email directly, view it on GitHub<#2805 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ASVVEUEO2NTZELUOIU2IUADX4VTALANCNFSM6AAAAAAZLC6PGA>. You are receiving this because you commented.Message ID: ***@***.***>