LogStream — High-Throughput Log Processing Pipeline

An end-to-end, fault-tolerant log ingestion and observability pipeline designed to handle massive concurrent traffic.

Built using an event-driven architecture, LogStream decouples ingestion, processing, storage, and visualization layers to ensure scalability, resilience, and near real-time observability under heavy load.

---

Features

• High-throughput asynchronous log ingestion
• Redis Streams-based event pipeline
• Consumer Group architecture for concurrent workers
• Real-time observability dashboard using WebSockets
• Infinite scrolling searchable log explorer
• SHA-256 based deduplication
• Batch database insertion using executemany
• Fault-tolerant replay handling using Redis Pending Entries List (PEL)
• PostgreSQL-backed persistent storage
• Full-text log searching
• Live ingestion velocity monitoring
• Horizontally scalable worker architecture

---

Architecture Overview

                ┌────────────────────┐
                │    Client Apps     │
                └─────────┬──────────┘
                          │
                          ▼
                ┌────────────────────┐
                │  FastAPI Ingestion │
                │      Layer         │
                └─────────┬──────────┘
                          │
                          ▼
                ┌────────────────────┐
                │   Redis Streams    │
                │  Consumer Groups   │
                └─────────┬──────────┘
                          │
              ┌───────────┴───────────┐
              ▼                       ▼
      ┌──────────────┐       ┌──────────────┐
      │ Worker #1    │       │ Worker #2    │
      │ asyncio      │       │ asyncio      │
      └──────┬───────┘       └──────┬───────┘
             │                      │
             └──────────┬───────────┘
                        ▼
              ┌────────────────────┐
              │    PostgreSQL      │
              │   Persistent DB    │
              └────────────────────┘

                        │
                        ▼
              ┌────────────────────┐
              │ Redis Pub/Sub      │
              │ Live Metrics Bus   │
              └─────────┬──────────┘
                        ▼
              ┌────────────────────┐
              │ React Dashboard    │
              │ WebSocket Client   │
              └────────────────────┘

---

Tech Stack

Backend

• FastAPI
• Python 3.11+
• SQLAlchemy
• asyncpg
• asyncio
• Redis Streams
• Redis Pub/Sub
• PostgreSQL
• Alembic
• uv

Frontend

• React
• TypeScript
• Tailwind CSS
• Recharts
• Lucide Icons

---

Key Engineering Decisions

Event-Driven Architecture

Instead of writing directly to PostgreSQL during ingestion, logs are first pushed into Redis Streams.

This decouples:

• API responsiveness
• database writes
• processing workload

Result:

• sub-millisecond ingestion latency
• higher throughput
• resilient backpressure handling

---

Idempotent Processing

Every log message is hashed using SHA-256 before insertion.

Combined with PostgreSQL:

ON CONFLICT DO NOTHING

workers can safely replay failed batches without corrupting the database.

This makes recovery from worker crashes deterministic and safe.

---

Batch Inserts Instead of Row Inserts

Workers collect logs in batches of 100 and use:

executemany()

instead of individual insert queries.

This reduces:

• database round trips
• transaction overhead
• disk sync pressure

Result:

• significantly higher throughput
• lower database CPU utilization

---

Consumer Groups for Horizontal Scaling

Redis Consumer Groups allow multiple workers to process logs concurrently without duplication.

Scaling becomes simple:

uv run processor.py

Run multiple instances across machines or containers.

---

Real-Time Observability Without DB Polling

Instead of continuously polling PostgreSQL for updates:

• workers publish lightweight metrics to Redis Pub/Sub
• FastAPI listens to Pub/Sub
• WebSocket broadcasts updates to connected clients

Result:

• near-zero database load during live monitoring
• efficient real-time dashboard streaming

---

Environment Variables

Create a .env file in the project root:

REDIS_HOST=localhost
REDIS_PORT=6379

DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres
DB_PASS=your_password
DB_NAME=logger

DATABASE_URL=postgresql://postgres:root@localhost:5432/logger

---

Backend Setup

Install uv

pip install uv

---

Create Virtual Environment

uv venv

Activate environment:

Linux/macOS

source .venv/bin/activate

Windows

.venv\Scripts\activate

---

Install Dependencies

If using pyproject.toml:

uv sync

---

Database Migrations (Alembic)

Initialize Alembic:

uv run alembic init alembic

Create migration:

uv run alembic revision --autogenerate -m "create logs table"

Run migrations:

uv run alembic upgrade head

---

Running the Backend

Start FastAPI Server

uv run uvicorn app.main:app --reload --port 8000

Start Worker

uv run processor.py

Run multiple workers in separate terminals for horizontal scaling.

---

Frontend Setup

Navigate into frontend directory:

npm install -g pnpm
cd client

Install dependencies:

pnpm install

Run development server:

pnpm run dev

---

Docker Setup (Recommended)

The easiest way to run LogStream is using Docker Compose. This starts all required services automatically, including PostgreSQL, Redis, the backend API, frontend, worker, and database migrations.

---

Prerequisites

• Docker
• Docker Compose

Verify your installation:

docker --version
docker compose version

---

Clone the Repository

git clone https://github.com/RahulGIT24/Scalable-Log-Processor
cd Scalable-Log-Processor

---

Create .docker.env

Inside the log_processor directory, create a file named:

log_processor/.docker.env

Add the following configuration:

REDIS_HOST=redis
REDIS_PORT=6379

DB_HOST=postgres
DB_PORT=5432
DB_USER=postgres
DB_PASS=root
DB_NAME=logger

DATABASE_URL=postgresql://postgres:root@postgres:5432/logger

TABLE_NAME=logs

Note: Inside Docker containers, services communicate using the Docker Compose service names (postgres, redis) rather than localhost.

---

Build the Images

docker compose build

---

Start the Application

docker compose up

Run in detached mode if preferred:

docker compose up -d

---

Services Started

Docker Compose automatically starts:

• PostgreSQL
• Redis
• FastAPI Backend
• React Frontend (served using Nginx)
• Background Log Processing Worker
• Alembic Migration Service

No additional setup is required.

---

Access the Application

Service	URL
Frontend	http://localhost:5173
Backend API	http://localhost:8000

---

Stop the Application

Stop all containers:

docker compose down

To also remove the PostgreSQL volume:

docker compose down -v

---

Rebuild After Dependency Changes

If Dockerfiles or dependencies change:

docker compose build --no-cache
docker compose up

---

Docker Architecture

Docker Compose runs the following services:

• frontend — React application served via Nginx
• backend — FastAPI REST API
• worker — Redis Streams consumer responsible for processing logs
• migrate — Alembic migration runner
• postgres — PostgreSQL database
• redis — Redis Streams and Pub/Sub server

Stress Testing

A stress testing script is included to simulate high concurrent traffic.

Run:

uv run stress_test.py

Default configuration:

• 500 concurrent connections
• 5000 log messages

Expected behavior:

• API accepts requests immediately
• Redis buffers ingestion spikes
• workers batch-process logs asynchronously
• dashboard displays live ingestion spikes in real time

---

API Endpoints

Ingestion

POST /api/ingest

Push a log into Redis Stream.

Example payload:

{
  "log_level": "ERROR",
  "message": "Database connection failed",
  "tags": ["DATABASE", "PROD"]
}

---

Analytics

GET /api/analytics/overview

Returns aggregated counts grouped by log level.

---

GET /api/analytics/search

Search historical logs with pagination.

Example:

/api/analytics/search?search=database&limit=50&offset=0

---

Real-Time Monitoring

WebSocket /api/analytics/ws

Streams:

• ingestion metrics
• live logs
• processing statistics

---

Frontend Dashboard Features

• Live WebSocket updates
• Infinite scroll log explorer
• Debounced full-text search
• Real-time ingestion velocity graph
• Log level aggregation cards
• Responsive observability UI
• Search without database polling

---

Demo Video

Enjoy Stress Testing LogStream in action:

Performance Notes

Under stress testing:

• ingestion remains non-blocking
• Redis absorbs traffic spikes
• workers process asynchronously
• PostgreSQL write amplification remains low due to batching

The architecture is designed around:

• backpressure tolerance
• fault recovery
• horizontal scalability
• observability-first design

---