LLM Chat Stream

EvilOllama - Secure LLM Server Setup for Cardputer

🧠 This guide shows how to set up a secure, authenticated Ollama LLM server on a Raspberry Pi 5 (8GB), publicly accessible and ready for M5Stack Cardputer.

---

✅ Prerequisites

• PC running Linux / WSL / Raspberry Pi 5 (8GB) ( or less but consider LLM need a lot of ressources)
• Domain name (e.g., 7h30th3r0n3.fr)
• Port 443 open to the internet (you can stay locally)
• M5Stack Cardputer (ESP32-S3)
• Wi-Fi access for Cardputer

---

🧰 1. Install Ollama

1. Install Ollama:

   curl -fsSL https://ollama.com/install.sh | sh

2. Pull a lightweight model:

   ollama pull tinyllama

3. Start the Ollama server:

   ollama serve &

4. Test it locally:

   curl -X POST http://localhost:11434/api/generate \
     -H "Content-Type: application/json" \
     -d '{"model": "tinyllama", "prompt": "Hello!", "stream": true}'

---

🔐 2. Install and Configure Apache for authenticated access

1. Install Apache and required modules:

   sudo apt install apache2 apache2-utils
   sudo a2enmod ssl proxy proxy_http headers auth_basic rewrite

2. Set up basic HTTP authentication:

sudo htpasswd -c /etc/apache2/.htpasswd theotherone

---

🌐 3. Enable SSL with Let's Encrypt

1. Install certbot:

sudo apt install certbot python3-certbot-apache

2. Run the setup:

sudo certbot --apache

---

🛡️ 4. Secure Reverse Proxy Configuration

1. Edit the main SSL virtual host:

sudo nano /etc/apache2/sites-available/000-default-le-ssl.conf

2. Add this inside <VirtualHost *:443>:

    # Redirect /evilOllama to Ollama locally
    RedirectMatch 301 ^/evilOllama$ /evilOllama/

    ProxyPass /evilOllama/ http://localhost:11434/
    ProxyPassReverse /evilOllama/ http://localhost:11434/

    <Location /evilOllama/>
        AuthType Basic
        AuthName "Evil Ollama Access"
        AuthUserFile /etc/apache2/.htpasswd
        Require valid-user
        RequestHeader set Authorization expr=%{HTTP:Authorization}
    </Location>

3. Reload Apache:

sudo systemctl reload apache2

---

🧪 5. Test the Secured Endpoint

curl -u userchoosen:YourPasswordHere \
  -X POST https://yourdomain.com/evilOllama/api/generate \
  -H "Content-Type: application/json" \
  -d '{"model": "tinyllama", "prompt": "Hello evil server!", "stream": true}'

✅ You should receive a JSON response from tinyllama.

---

🎉 Ready for Cardputer!

You can now enter all the informations in config.txt file in config folder, restart the carputer and enjoy !