posts/ai-stack-tutorial/

[giscus[bot]]

posts/ai-stack-tutorial/

In this tutorial we’ll walk through my local, private, self-hosted AI stack so that you can run it too. If you’re looking for the overview of this stack, you can out the video here Self-Hosted AI That’s Actually Useful

https://technotim.live/posts/ai-stack-tutorial/

[l33tlamer]

Thanks for sharing Tim!

Quick note, i was unable to get OpenWebUI working with searxng, even tho the containers were able to connect (docker exec into webui, curl the searxng, no problems). All i got in the WebUI was a 403. Tried it with and without the search?q= and always the same result.

The fix for me was to edit the searxng settings.yml file and under formats, add json as a option (only html is there by default). And then it instantly works.

This is also mentioned in the OpenWebUI documentation to enable searxng integratation:

https://docs.openwebui.com/tutorial/web_search

[oliverbarreto]

I can't get it to work with searxng... I've edited settings.yml file to include formats: [html, json] and still doesn't work... Am I missing something ???

After enabling JSON, I can don't receive the 403 error anymore when making a direct call on the browser. It simply returns a valid JSON response with the results of the search query, but OpenWebUI still spits the error with too many redirects and 403...

Can you please provide more details about this configuration ...

In the mean time I got web search working with DuckDuckGo which require no configuration... but I NEED privacy provided by Searxng

[frostronic]

what fixed it for me, in addition to @l33tlamer's suggestions above, was to expose the internal container port in the stack, e.g.:

ports:
  - 8080

[hanley-development]

note to self check the comments section when I hit an error. After i added - json to the settings.yml and restarted the websearch worked.

open-webui/open-webui#2717 (comment)

[taker218]

I already mentioned it in the twitch chat yesterday, but got it all running now (hence the home assistant integration, need to do that next).
It seems like the Quadro M4000 I have is not compatible with Cuda 12.4, which is needed for GPU transcription with whisper. But CPU transcription is working just fine (even though it might be a bit slower)
Also had kind of the same problem with the libretranslate-container, which also requires Cuda 12.4. But I got it working with the image libretranslate:v1.5.7-cuda, which seems to be the last image to not use Cuda 12.4.
Maybe this might help others

[zehndi77]

for me the ollama container would not want to use the graphics card, with an error "CPU does not have minimum vector extensions ...." based on AVX missing..
I had to set the CPU type in the proxmox config of the docker host to "host" then it would work.

Thank you Tim for this tutorial!

[timothystewart6]

Good call! Yes, forgot to mention it but that's always my choice for CPU type! Thank you!

[hanley-development]

Thanks Tim for this tutorial! Biggest take away... aside from the fact we now have local AI setup.... btop! I remember when htop was new and cool! I just set it up for gpu stats and my mind was blown. Actually that may make for good content: local system and collection of stats.

[robflate]

Thanks for the amazing resource. You seem to use the following environment variables everywhere;

- PUID=${PUID:-1000}
- PGID=${PGID:-1000}

Are you sure those are supported environment variables in these images? I can't find any mention of them in either the SearXNG or Open WebUI repos.

I try to run most containers as a non-root user but both SearXNG and Open WebUI both create files as root despite using these variables so just wanted to check. Thanks again.

[garygentry]

Awesome job as always! Maybe a dumb question, but where is the Traefik server running in this setup? Does it have to be setup on the same machine and using the same external Docker network? I got the Traefik + Wildcard certs container setup and running (another awesome video!) but now scratching my head a little for how to tie these together..

[TheWolfOfWalmart]

Also confused about this. I've tried implementing docker swarm so the AI stack could connect to traefik on the docker host I made following the other tutorial but was not successful. Did you figure it out?

[garygentry]

Never really found a definitive answer but ended up running Traefik in a single container and defining all rules for all of the services from external hosts in the traefik config files (instead of tags in the docker config). Then configure DNS to route all custom names to this Traefik server.

It seems like it should be possible to get everything running on the same docker network using swarm to be able to define rules in the docker tags, but have never tried to setup swarm and didn't want to mess with it just to support this.

Would love to learn if there's a better way to do it and keep the rules close to the container definitions (though after doing it this way, I can see some advantage to having rules centralized in the traefik config vs scattered across several docker compose files. My setup is simple so probably other trade offs). Open to being schooled on all of this!

[USNBALLER]

Awesome job on this guide!

I was able to get most of this up and running, but I noticed this guide wasn’t as detailed as your Traefik guide. Would you consider doing a follow-up or providing additional details, especially on pulling models and linking them to the Docker instance? For example, instructions on how to get the Llama files and configure them to run in Docker from the /opt/stacks/ai-stack/ollama/models directory.

Additionally, I noticed that if you followed the Traefik guide, the network should be set to proxy, unless there’s another reason for adding a new network that I might not be understanding. Example:

yaml
Copy code
networks:
proxy:
external: true

traefik:

external: true

Would love your insights or updates to make it clearer for everyone. Thanks again for the excellent work!

[USNBALLER]

Sorry for the weird format above, lol.