Open Zephyr:

I know LLM's are cool and all, but you know whats not cool? Setting up the Infrastructure. Well look no further, here are some scripts and ways to deploy Containerized LLM's in a flash.

What it Deploys:

Currently it deploys OpenWebUI, with ollama API, Langfuse Tracing, and ChromaDB - All hosted in Docker - I would reccomend reviewing the Docker Compose to really adjust settings, and make it production ready, then all you need to do is run the setup.sh and you should be golden.

Whats Next:

Automating the deployment of also the iac code, and making it more modular, so you can deploy it on any cloud provider, and any instance type.

Please Review the Terraform Code before using it, currently using Bash Variable and only supports utilizng EC2 instance, with a T4

How it works.

1. We Have some Infra code, located in the iac dir.

2. packer holds code for making infra templates.

3. terraform holds code for deploying those infra templates.

4. cfg holds scripts for configuring the infra.

5. We have the application, which is hosted in Docker. Everything you need, Ollama API, Langfuse Tracing, OpenWebUI, ChromaDB, and all the extras, all hosted in Docker. I would reccomend reviewing the Docker Compose to really adjust settings, and make it production ready, then all you need to do is run the setup.sh and you should be golden.

Utilizing:

• Langfuse
  • OpenWebUI Langfuse Pipeline
• OpenWebUI Github.
• ChromaDB Github.
• Ollama Github
• Terraform
• Packer
• Docker

Example Trace:

Feel free to reach out, if any questions!

• If this gets attention well, more documentation will be added, of course better automation to come.