Raspberry Pi 5 + Hailo AI HAT+ 2 LLM Setup

Run local LLM inference on a Raspberry Pi 5 using the Hailo AI HAT+ 2 accelerator (40 TOPS @ INT4).

What's Included

File	Description
docs/raspberry-pi-hailo-hat2-llm-setup.md	Complete setup guide from bare metal to working LLM
docker-compose.yml	Open WebUI container configuration
manage-hailo.sh	Unified management script for all services
hailo-ollama.service	Systemd unit for auto-start on boot

Hardware Requirements

• Raspberry Pi 5 (8GB recommended)
• Hailo AI HAT+ 2 (8GB onboard memory)
• MicroSD card (32GB+)
• Power supply (27W USB-C recommended)

View hardware setup

Quick Start

After completing the setup guide:

# Start hailo-ollama and Open WebUI
./manage-hailo.sh start

# Check status
./manage-hailo.sh status

# Access the web UI
# http://<your-pi-ip>:3000

Available Models

The Hailo GenAI Model Zoo includes:

• llama3.2:3b
• deepseek_r1_distill_qwen:1.5b
• qwen2.5-coder:1.5b
• qwen2.5-instruct:1.5b
• qwen2:1.5b

Models are pre-compiled HEF binaries optimized for the Hailo accelerator.

Architecture

┌─────────────────────────────────────────────────────────┐
│                    Raspberry Pi 5                       │
│  ┌─────────────────┐    ┌─────────────────────────────┐ │
│  │   Open WebUI    │───▶│      hailo-ollama           │ │
│  │   (Docker)      │    │   (native, port 8000)       │ │
│  │   port 3000     │    │                             │ │
│  └─────────────────┘    └──────────────┬──────────────┘ │
│                                        │                │
│                         ┌──────────────▼──────────────┐ │
│                         │     Hailo AI HAT+ 2         │ │
│                         │   40 TOPS @ INT4            │ │
│                         │   8GB onboard memory        │ │
│                         └─────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘

Management Commands

./manage-hailo.sh start         # Start all services
./manage-hailo.sh stop          # Stop all services
./manage-hailo.sh restart       # Restart all services
./manage-hailo.sh status        # Show status

./manage-hailo.sh start-hailo   # Start only hailo-ollama
./manage-hailo.sh start-webui   # Start only Open WebUI

./manage-hailo.sh logs-hailo    # Follow hailo-ollama logs
./manage-hailo.sh logs-webui    # Follow Open WebUI logs

./manage-hailo.sh pull <model>  # Pull a model
./manage-hailo.sh chat          # Quick CLI chat test

API Endpoints

The hailo-ollama server exposes an Ollama-compatible API:

Endpoint	Description
GET /hailo/v1/list	List available models
POST /api/pull	Pull/download a model
POST /api/chat	Chat completion (streaming)

Notes

• This is not standard Ollama - it's Hailo's API-compatible server
• Models are HEF binaries, not GGUF files
• The upstream ollama CLI won't work directly with this setup
• Inference runs entirely on the Hailo accelerator

License

MIT