GenAI Application Demo using Model Runner

A modern chat application demonstrating integration of frontend technologies with local Large Language Models (LLMs).

Overview

This project is a full-stack GenAI chat application that showcases how to build a Generative AI interface with a React frontend and Go backend using Model Runner.

Two Methods

There are two ways you can use Model Runner:

Using Internal DNS
Using TCP

Using Docker socket

This methods points to the same Model Runner (llama.cpp engine) but through different connection method. It uses the internal Docker DNS resolution (model-runner.docker.internal)

Architecture

The application consists of three main components:

Frontend: React TypeScript application providing a responsive chat interface
Backend: Go server that handles API requests and connects to the LLM
Model Runner: Llama 3.2 (1B parameter) model

┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│   Frontend  │ >>> │   Backend   │ >>> │Model Runner │
│  (React/TS) │     │    (Go)     │     │  (Llama 3.2)│
└─────────────┘     └─────────────┘     └─────────────┘
      :3000              :8080             :12434

Features

Real-time chat interface with message history
Streaming AI responses (tokens appear as they're generated)
Dockerized deployment for easy setup
Local LLM integration (no cloud API dependencies)
Cross-origin resource sharing (CORS) enabled
Comprehensive integration tests using Testcontainers

Prerequisites

Docker and Docker Compose
Git
Go 1.19 or higher
Download the model before proceeding further

docker model pull ai/llama3.2:1B-Q8_0

Quick Start

Clone this repository:

git clone https://github.com/ajeetraina/genai-app-demo.git
cd genai-app-demo

Start the application using Docker Compose:
```
docker compose up -d -build
```
Access the frontend at http://localhost:3000

Development Setup

Frontend

The frontend is a React TypeScript application using Vite:

cd frontend
npm install
npm run dev

Backend

The Go backend can be run directly:

go mod download
go run main.go

Make sure to set the environment variables in backend.env or provide them directly.

Using TCP Host

This methods points to the same Model Runner (llama.cpp engine) but through different connection method. It uses the host-side TCP support via host.docker.internal:12434

Configuration

The backend connects to the LLM service using environment variables defined in backend.env:

BASE_URL: URL for the model runner
MODEL: Model identifier to use
API_KEY: API key for authentication

Deployment

The application is configured for easy deployment using Docker Compose. See the compose.yaml file for details.

License

MIT

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

dockersamples/genai-app-demo

ErrorLooks like something went wrong!

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Error
Looks like something went wrong!

Packages