Distributed Webhook Event Processing System

A full-stack reliability simulator that demonstrates how production webhook infrastructure handles the hard parts — duplicate deliveries, signature verification, retry logic, dead-letter queues, and real-time observability — all in one runnable system.

The backend processes webhooks through a distributed queue and streams every state change live to the frontend dashboard over WebSocket.

---

The Problem It Solves

Webhooks are deceptively simple to receive but surprisingly hard to make reliable. At scale, you deal with:

• Duplicate deliveries — the sender retries because they didn't get a 200 in time
• Replay attacks — captured requests replayed hours later
• Processing failures — third-party services go down mid-processing
• Lost events — worker crashes between receiving and processing
• No visibility — you have no idea what's happening inside the queue

This system implements the standard solutions to each of these and makes them observable through a live dashboard.

---

Architecture

┌─────────────────────────────────────────────────────────┐
│                        Frontend                         │
│         React · TanStack Query · Zustand · Recharts     │
│                                                         │
│   Metrics Bar · Live Event Feed · Throughput Chart      │
│   Dead Letter Panel · Scenario Controls · Toasts        │
└──────────────────────┬──────────────────────────────────┘
                       │  HTTP (axios)   WebSocket
                       │
┌──────────────────────▼──────────────────────────────────┐
│                       Backend                           │
│                FastAPI  (async, Python 3.12)            │
│                                                         │
│  POST /webhook                                          │
│   ├── HMAC-SHA256 signature verification                │
│   ├── Replay attack prevention (5-min timestamp window) │
│   ├── Redis idempotency lock (SET NX EX, atomic)        │
│   └── Enqueue to Celery ─────────────────────┐          │
│                                              │          │
│  GET  /ws  ◄── Redis Pub/Sub ◄──────────────┘          │
│              (ws:events channel)                        │
└──────────────────────┬──────────────────────────────────┘
                       │
          ┌────────────┼────────────┐
          │            │            │
     ┌────▼────┐  ┌────▼────┐  ┌───▼──────┐
     │  Redis  │  │Postgres │  │  Celery  │
     │ broker  │  │  event  │  │  worker  │
     │ pub/sub │  │   log   │  │ (sync)   │
     │idempot. │  │         │  │ retries  │
     └─────────┘  └─────────┘  └──────────┘

---

Event Lifecycle

Every webhook goes through this pipeline:

QUEUED → PROCESSING → COMPLETED
                    ↘ RETRYING  (up to 3×, 2^n second backoff)
                              ↘ DEAD_LETTERED  (replayable from dashboard)

Rejected webhooks (bad signature, expired timestamp, duplicate) appear as FAILED or DUPLICATE in the dashboard — the sender always gets HTTP 200 regardless.

---

Simulation Scenarios

The dashboard includes four built-in scenarios you can trigger with one click:

Scenario	What it demonstrates
Duplicate Delivery	Idempotency — same event sent twice, second is silently dropped
Dead Letter	Retry exhaustion — event fails 3 times and lands in the DLQ
Signature Attack	Security — invalid HMAC rejected without exposing why
Campaign Blast	Throughput — 50 concurrent events processed asynchronously

---

Stack

Backend — FastAPI · Celery · Redis · PostgreSQL · SQLAlchemy 2.0 · Alembic · structlog · Docker

Frontend — React 19 · Vite · TanStack Query v5 · Zustand v5 · axios · Recharts · sonner · Tailwind CSS

---

Quick Start

1. Start the backend (API + Celery worker + Redis):

cd backend
cp .env.example .env        # fill in your values
docker compose up -d
docker compose exec api alembic upgrade head   # first time only

2. Start the frontend:

cd frontend
docker compose up -d

Or without Docker:

cd frontend
npm install && npm run dev

Open http://localhost:5173. The backend API is at http://localhost:8000/docs.

---

Repository Layout

├── backend/               # FastAPI + Celery service
│   ├── app/
│   │   ├── api/routes/    # One file per resource
│   │   ├── core/          # Config, enums, exceptions
│   │   ├── infrastructure/# DB, Redis, Celery setup
│   │   ├── models/        # SQLAlchemy models
│   │   ├── services/      # Business logic
│   │   ├── tasks/         # Celery task
│   │   └── websocket/     # WS manager + Redis subscriber
│   ├── alembic/           # Migrations
│   ├── docker/            # Dockerfile
│   ├── docker-compose.yml
│   └── README.md          # Backend deep-dive
│
└── frontend/              # React dashboard
    ├── src/
    │   ├── components/    # UI components by feature
    │   ├── hooks/         # TanStack Query + WebSocket hooks
    │   ├── services/      # axios instance + API wrappers
    │   └── stores/        # Zustand store (real-time state only)
    ├── Dockerfile
    ├── docker-compose.yml
    └── README.md          # Frontend deep-dive

For implementation details, see the individual READMEs:

• backend/README.md — reliability patterns, API reference, migrations
• frontend/README.md — state management architecture, key design decisions