v1.0.0: Production Private LLM Chat on AWS SageMaker with vLLM

🚀 What's New in v1.0.0

Welcome to the major v1.0.0 stable release of Private LLM Chat on AWS SageMaker!

🌟 Release Highlights

• Private Self-Hosted ChatGPT: Host and run open-source LLMs (Qwen, Llama 3, Gemma, DeepSeek) inside your private AWS VPC without third-party data leakage.
• Cost-Optimized GPU Automation: SageMaker endpoints start automatically when users chat and self-terminate after 30 minutes of idle time.
• Enterprise Features: Multi-user accounts with Google OAuth, TOTP two-factor authentication, session management, and document attachment processing.
• Integrated Web Search & Voice Input: Self-hosted SearXNG search integration providing cited web sources, and local Whisper.cpp voice transcription.
• Full-Stack Infrastructure as Code: Modular Terraform provisioning ECS Fargate, CloudFront, ALB, RDS PostgreSQL, S3, and IAM policies.

v0.1.0: First public release

Your own ChatGPT, on your own GPU, in your own AWS account.

Deploy open-source LLMs (Qwen, Llama, Gemma, DeepSeek…) on Amazon SageMaker with vLLM, and put a ChatGPT-style app in front of them. The model, the database and the files stay in your AWS account, and GPU endpoints switch themselves off when nobody is using them.

What's included

• Chat: streamed answers, history with search, rename and pin, regenerate and stop, Markdown with code, tables and math, custom instructions, model picker, dark mode and a mobile layout
• Documents and images: attach PDFs, Word files, text and code; images go straight to vision models
• OCR: extract text from photos, screenshots and scanned PDFs with a vision model, or with Tesseract on the CPU
• Accounts: email and password with verification, password reset, Sign in with Google, and two-factor auth (authenticator app or email codes) with recovery codes
• Models: SageMaker endpoints on AWS's vLLM container, any OpenAI-compatible server (vLLM, Ollama, LM Studio) and a mock for development, with ready-made presets from Qwen3-VL 4B to Llama 3.3 70B
• GPU control: each model is on_demand (starts on the first message, deleted after 30 idle minutes), always_on, off or manual; several models can share one GPU through the included vLLM router
• AWS infrastructure in Terraform: CloudFront (HTTPS), ALB, ECS Fargate (ARM), RDS PostgreSQL, private S3, SES and SageMaker, deployed from GitHub Actions over OIDC with no stored AWS keys
• Local development: make up runs the whole stack in Docker, with Mailpit for emails and an optional Ollama profile

Get started

Five minutes, no GPU, just Docker:

git clone https://github.com/codewithmuh/private-llm-chat-sagemaker-aws.git
cd private-llm-chat-sagemaker-aws
make up

Open http://localhost:3000 and sign up. The verification code arrives in Mailpit at http://localhost:8025.

Then go as far as you need:

1. Run it locally: free
2. A real model on your machine with Ollama: free
3. Your own GPU on SageMaker: about $1.13–2.61 per GPU hour
4. The whole app on AWS: about $80–90 a month, plus GPU hours
5. Advanced: weights in S3, several models per GPU, 70B models

What it costs

On AWS, the app itself costs about $80–90 a month in us-east-1. GPUs are billed by the hour for as long as an endpoint exists: an ml.g6e.xlarge is $2.61/hour, about $1,900 a month if left running. With on_demand, one person chatting an hour a day pays about $133 a month for the GPU. The full breakdown is in docs/cost.md.

Warning

A forgotten GPU costs about $63 a day. ./scripts/gpu.sh status shows what is running.

Not in this release yet

RAG over your documents, an Amazon Bedrock provider, editing a sent message and sharing conversations are next on the roadmap. Bug reports and ideas are welcome in Issues.

MIT licensed.