Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
title: Overview
title: Run LLMs locally on Raspberry Pi 5 for Edge AI

weight: 2

### FIXED, DO NOT MODIFY
Expand All @@ -8,66 +9,65 @@ layout: learningpathall

## Overview

This Learning Path walks you through deploying an efficient large language model (LLM) locally on the Raspberry Pi 5, powered by an Arm Cortex-A76 CPU. This will allow you to control your smart home using natural language, without relying on cloud services. With rapid advances in Generative AI and the power of Arm Cortex-A processors, you can now run advanced language models directly in your home on the Raspberry Pi 5.
This Learning Path walks you through deploying an efficient large language model (LLM) locally on the Raspberry Pi 5, powered by an Arm Cortex-A76 CPU. This setup enables you to control your smart home using natural language without relying on cloud services. With rapid advances in generative AI and the power of Arm Cortex-A processors, you can now run advanced language models directly in your home on the Raspberry Pi 5.

You will create a fully local, privacy-first smart home system that leverages the strengths of Arm Cortex-A architecture. The system can achieve 15+ tokens per second inference speeds using optimized models like TinyLlama and Qwen, while maintaining the energy efficiency that makes Arm processors a good fit for always-on applications.
You will create a fully local, privacy-first smart home system that leverages the strengths of Arm Cortex-A architecture. The system can achieve 15+ tokens per second inference speeds using optimized models like TinyLlama and Qwen, while maintaining the energy efficiency that makes Arm processors well suited for always-on applications.

## Why Arm Cortex-A for Edge AI?
## Why Arm Cortex-A76 makes Raspberry Pi 5 ideal for Edge AI

The Raspberry Pi 5's Arm Cortex-A76 processor can manage high-performance computing tasks like AI inference. Key architectural features include:

- The **superscalar architecture** allows the processor to execute multiple instructions in parallel, improving throughput for compute-heavy tasks.
- **128-bit NEON SIMD support** accelerates matrix and vector operations, which are common in the inner loops of language model inference.
- The **multi-level cache hierarchy** helps reduce memory latency and improves data access efficiency during runtime.
- The **thermal efficiency** enables sustained performance without active cooling, making it ideal for compact or always-on smart home setups.
- **Superscalar architecture**: Executes multiple instructions in parallel, improving throughput for compute-heavy tasks
- **128-bit NEON SIMD support**: Accelerates matrix and vector operations, common in the inner loops of language model inference
- **Multi-level cache hierarchy**: Reduces memory latency and improves data access efficiency during runtime
- **Thermal efficiency**: Enables sustained performance without active cooling, making it ideal for compact or always-on smart home setups

These characteristics make the Raspberry Pi 5 well-suited for workloads like smart home assistants, where responsiveness, efficiency, and local processing are important. Running LLMs locally on Arm-based devices brings several practical benefits. Privacy is preserved, since conversations and routines never leave the device. With optimized inference, the system can offer responsiveness under 100 ms, even on resource-constrained hardware. It remains fully functional in offline scenarios, continuing to operate when internet access is unavailable. Developers also gain flexibility to customize models and automations. Additionally, software updates and an active ecosystem continue to improve performance over time.
These characteristics make the Raspberry Pi 5 well suited for workloads like smart home assistants, where responsiveness, efficiency, and local processing are important. Running LLMs locally on Arm-based devices brings several practical benefits. Privacy is preserved, since conversations and routines never leave the device. With optimized inference, the system can offer responsiveness under 100 ms, even on resource-constrained hardware. It remains fully functional in offline scenarios, continuing to operate when internet access is unavailable. Developers also gain flexibility to customize models and automations. Additionally, software updates and an active ecosystem continue to improve performance over time.

## Arm Ecosystem Advantages
## Leverage the Arm ecosystem for Raspberry Pi Edge AI

For the stack in this setup, Raspberry Pi 5 benefits from the extensive developer ecosystem:

- Optimized compilers including GCC and Clang with Arm-specific enhancements
- Native libraries such as gpiozero and lgpio are optimized for Raspberry Pi
- Community support from open-source projects where developers are contributing Arm-optimized code
- Arm maintains a strong focus on backward compatibility, which reduces friction when updating kernels or deploying across multiple Arm platforms
- Community support from open-source projects where developers contribute Arm-optimized code
- Backward compatibility in Arm architecture reduces friction when updating kernels or deploying across platforms
- The same architecture powers smartphones, embedded controllers, edge devices, and cloud infrastructure—enabling consistent development practices across domains

## Performance Benchmarks on Raspberry Pi 5
## Performance benchmarks on Raspberry Pi 5

The table below shows inference performance for several quantized models running on a Raspberry Pi 5. Measurements reflect single-threaded CPU inference with typical prompt lengths and temperature settings suitable for command-based interaction.

| Model | Tokens/Sec | Avg Latency (ms) |
| Model | Tokens/sec | Avg latency (ms) |
| ------------------- | ---------- | ---------------- |
| qwen:0.5b | 17.0 | 8,217 |
| tinyllama:1.1b | 12.3 | 9,429 |
| deepseek-coder:1.3b | 7.3 | 22,503 |
| gemma2:2b | 4.1 | 23,758 |
| deepseek-r1:7b | 1.6 | 64,797 |

## LLM benchmark insights on Raspberry Pi 5

What does this table tell us? Here are some performance insights:

- Qwen 0.5B and TinyLlama 1.1B deliver fast token generation and low average latency, making them suitable for real-time interactions like voice-controlled smart home commands.
- DeepSeek-Coder 1.3B and Gemma 2B trade off some speed for improved language understanding, which can be useful for more complex task execution or context-aware prompts.
- DeepSeek-R1 7B offers advanced reasoning capabilities with acceptable latency, which may be viable for offline summarization, planning, or low-frequency tasks.
- Qwen 0.5B and TinyLlama 1.1B deliver fast token generation and low average latency, making them suitable for real-time interactions such as voice-controlled smart home commands
- DeepSeek-Coder 1.3B and Gemma 2B trade some speed for improved language understanding, which can be useful for complex tasks or context-aware prompts
- DeepSeek-R1 7B offers advanced reasoning capabilities with acceptable latency, which may be viable for offline summarization, planning, or low-frequency tasks

## Supported Arm-Powered Devices
## Supported Arm-powered devices

This Learning Path focuses on the Raspberry Pi 5, but you can adapt the concepts and code to other Arm-powered devices:
This Learning Path focuses on the Raspberry Pi 5, but you can adapt the concepts and code to other Arm-powered devices.

### Recommended Platforms
## Recommended platforms

| Platform | CPU | RAM | GPIO Support | Model Size Suitability |
|------------------|----------------------------------|----------------|-------------------------------|-----------------------------|
| **Raspberry Pi 5** | Arm Cortex-A76 quad-core @ 2.4GHz | Up to 16GB | Native `lgpio` (high-performance) | Large models (8–16GB) |
| **Raspberry Pi 4** | Arm Cortex-A72 quad-core @ 1.8GHz | Up to 8GB | Compatible with `gpiozero` | Small to mid-size models |
| **Other Arm Devices** | Arm Cortex-A | 4GB min (8GB+ recommended) | Requires physical GPIO pins | Varies by RAM |
| Platform | CPU | RAM | GPIO support | Model size suitability |
| ------------------- | -------------------------------- | -------------- | ------------------------------ | --------------------------- |
| **Raspberry Pi 5** | Arm Cortex-A76 quad-core @ 2.4GHz | Up to 16GB | Native `lgpio` (high-performance) | Large models (8–16GB) |
| **Raspberry Pi 4** | Arm Cortex-A72 quad-core @ 1.8GHz | Up to 8GB | Compatible with `gpiozero` | Small to mid-size models |
| **Other Arm devices** | Arm Cortex-A | 4GB min (8GB+ recommended) | Requires physical GPIO pins | Varies by RAM |

Additionally, the platform must:
Additionally, the platform must meet the following requirements:

- GPIO pins available for hardware control
- Use Python 3.8 or newer
- Python 3.8 or newer
- Ability to run [Ollama](https://ollama.com/)

Continue to the next section to start building a smart home system that highlights how Arm-based processors can enable efficient, responsive, and private AI applications at the edge.
In the next section, you’ll set up the software dependencies needed to start building your privacy-first smart home system on Raspberry Pi 5.
Original file line number Diff line number Diff line change
@@ -1,63 +1,67 @@
---
title: Set up software dependencies
title: Set up software dependencies on Raspberry Pi 5 for Ollama and LLMs
weight: 3

### FIXED, DO NOT MODIFY
layout: learningpathall
---

## Overview

In this section, you’ll prepare your Raspberry Pi 5 by installing Python, required libraries, and Ollama, so you can run large language models (LLMs) locally.

{{% notice Note %}}
This guide assumes you have set up your Raspberry Pi with Raspberry Pi OS and network connectivity. For Raspberry Pi 5 setup help, see: [Raspberry Pi Getting Started](https://www.raspberrypi.com/documentation/)
This Learning Path assumes you have set up your Raspberry Pi with Raspberry Pi OS and network connectivity. For Raspberry Pi 5 setup support, see [Raspberry Pi Getting Started](https://www.raspberrypi.com/documentation/).
{{% /notice %}}

## Connect to Your Raspberry Pi 5
## Connect to your Raspberry Pi 5

### Option 1: Using a display
### Option 1: Use a display

The easiest way to work on your Raspberry Pi is connecting it to an external display through one of the micro HDMI ports. This setup also requires a keyboard and mouse to navigate.
The easiest way to work on your Raspberry Pi is by connecting it to an external display through one of the microHDMI ports. This setup also requires a keyboard and mouse.

### Option 2: Using SSH
### Option 2: Use SSH

You can also use SSH to access the terminal. To use this approach you need to know the IP address of your device. Ensure your Raspberry Pi 5 connects to the same network as your host computer. Access your device remotely via SSH using the terminal or any SSH client.
You can also use SSH to access the terminal. To use this approach, you need to know the IP address of your device. Ensure your Raspberry Pi 5 is on the same network as your host computer. Access your device remotely via SSH using the terminal or any SSH client.

Replace `<user>` with your Pi's username (typically `pi`), and `<pi-ip>` with your Raspberry Pi 5's IP address.

```bash
ssh <user>@<pi-ip>
```

## Set up the dependencies
## Install Python and system dependencies

Create a directory called `smart-home` in your home directory and navigate into it:

```bash
mkdir $HOME/smart-home
cd $HOME/smart-home
mkdir -p "$HOME/smart-home"
cd "$HOME/smart-home"
```

The Raspberry Pi 5 includes Python 3 pre-installed, but you need additional packages:
The Raspberry Pi 5 includes Python 3 preinstalled, but you need additional packages:

```bash
sudo apt update && sudo apt upgrade
sudo apt install python3 python3-pip python3-venv git curl build-essential gcc python3-lgpio
sudo apt update && sudo apt upgrade -y
sudo apt install -y python3 python3-pip python3-venv git curl build-essential gcc python3-lgpio
```

### Configure the virtual environment
## Configure a virtual environment

The next step is to create and activate a Python virtual environment. This approach keeps project dependencies isolated and prevents conflicts with system-wide packages:
Create and activate a Python virtual environment to isolate project dependencies:

```bash
python3 -m venv venv
source venv/bin/activate
```

Install all required libraries and dependencies:
Install the required libraries:

```bash
pip install ollama gpiozero lgpio psutil httpx orjson numpy fastapi uvicorn uvloop numpy
pip install ollama gpiozero lgpio psutil httpx orjson numpy fastapi uvicorn uvloop
```

### Install Ollama
## Install Ollama

Install Ollama using the official installation script for Linux:

Expand All @@ -70,27 +74,29 @@ Verify the installation:
```bash
ollama --version
```
If installation was successful, the output from the command should match that below.

If installation was successful, the output should be similar to:

```output
ollama version is 0.11.4
```

## Download and Test a Language Model
## Run a test LLM with Ollama on Raspberry Pi 5

Ollama supports various models. This guide uses deepseek-r1:7b as an example, but you can also use `tinyllama:1.1b`, `qwen:0.5b`, `gemma2:2b`, or `deepseek-coder:1.3b`.
Ollama supports various models. This guide uses `deepseek-r1:7b` as an example, but you can also use `tinyllama:1.1b`, `qwen:0.5b`, `gemma2:2b`, or `deepseek-coder:1.3b`.

The `run` command will set up the model automatically. You will see download progress in the terminal, followed by the interactive prompt when ready.
The `run` command sets up the model automatically. You will see download progress in the terminal, followed by an interactive prompt when ready.

```bash
ollama run deepseek-r1:7b
```

{{% notice Troubleshooting %}}
If you run into issues with the model download, here are some things to check:
If you run into issues with the model download, try the following:

- Confirm internet access and sufficient storage space on your microSD card
- Try downloading smaller models like `qwen:0.5b` or `tinyllama:1.1b` if you encounter memory issues. 16 GB of RAM is sufficient for running smaller to medium-sized language models. Very large models may require more memory or run slower.
- Clear storage or connect to a more stable network if errors occur
- Confirm internet access and sufficient storage space on your microSD card.
- Try smaller models like `qwen:0.5b` or `tinyllama:1.1b` if you encounter memory issues. 16 GB of RAM is sufficient for small to medium models; very large models may require more memory or run slower.
- Clear storage or connect to a more stable network if errors occur.
{{% /notice %}}

With the model set up through `ollama`, move on to the next section to start configuring the hardware.
With the model set up through Ollama, move on to the next section to start configuring the hardware.
Original file line number Diff line number Diff line change
@@ -1,18 +1,22 @@
---
title: Test GPIO pins
title: Test Raspberry Pi 5 GPIO pins for smart home devices
weight: 4

### FIXED, DO NOT MODIFY
layout: learningpathall
---

The next step is to test the GPIO functionality. In this section, you will configure a LED light to simulate a smart-home device.
## Overview

## Verify GPIO Functionality
The next step is to test the GPIO functionality. In this section, you configure an LED light to simulate a smart home device.

Bring out your electronics components. Connect the anode (long leg) of an LED in series with a 220Ω resistor to GPIO 17 (physical pin 11). Connect the cathode (short leg) to a ground (GND) pin. See image below for the full setup:
## Verify GPIO setup on Raspberry Pi 5

![Raspberry Pi connected to a breadboard with a green LED and jumper wires](pin_layout.jpg "Raspberry Pi connected to a breadboard with a green LED and jumper wires")
Gather your electronic components. Connect the anode (long leg) of an LED in series with a 220Ω resistor to GPIO 17 (physical pin 11). Connect the cathode (short leg) to a ground (GND) pin.

See the image below for the full setup:

![Raspberry Pi connected to a breadboard with a green LED and jumper wires alt-text#center](pin_layout.jpg "Raspberry Pi connected to a breadboard with a green LED and jumper wires")

Create a Python script named `testgpio.py`:

Expand All @@ -21,7 +25,7 @@ cd $HOME/smart-home
vim testgpio.py
```

Copy this code into the file:
Add the following code to the file:

```python
#!/usr/bin/env python3
Expand All @@ -32,7 +36,7 @@ from gpiozero.pins.lgpio import LGPIOFactory
# Set lgpio backend for Raspberry Pi 5
Device.pin_factory = LGPIOFactory()

# Setup GPIO pin 17
# Set up GPIO pin 17
pin1 = LED(17)

try:
Expand All @@ -52,19 +56,20 @@ python testgpio.py
The LED should blink every two seconds. If you observe this behavior, your GPIO setup works correctly.

{{% notice Troubleshooting %}}
If you run into issues with the hardware setup, here are some things to check:
- Try fixing missing dependencies by running the following command:
```bash
sudo apt-get install -f
```
- If you're running into GPIO permission issues, run Python scripts with `sudo` or add your user to the `gpio` group. Don't forget to log out for the changes to take effect.
```bash
sudo usermod -a -G gpio $USER
```
If you run into issues with the hardware setup, check the following:

- Fix missing dependencies with:
```bash
sudo apt-get install -f
```
- If you encounter GPIO permission issues, run Python scripts with `sudo` or add your user to the `gpio` group. Don’t forget to log out for the changes to take effect:
```bash
sudo usermod -a -G gpio $USER
```
- Double-check wiring and pin numbers using the Raspberry Pi 5 pinout diagram
- Ensure proper LED and resistor connections
- Verify GPIO enablement in `raspi-config` if needed
- Use a high-quality power supply
{{% /notice %}}

With a way to control devices using GPIO pins, you can move on to the next section to interact with them using language models and the user interface.
With GPIO pins working, you can now move on to the next section to interact with devices using language models and the user interface.
Loading