Event-Driven Manufacturing Microservices with Redpanda Connect

A complete, runnable pipeline that ingests simulated factory sensor data via MQTT, routes it through Redpanda Connect into domain-specific Redpanda topics, and processes it with three independent Python microservices.

What This Project Does

1. A sensor simulator publishes temperature, vibration, and pressure telemetry for three machines over MQTT.
2. Redpanda Connect subscribes to the MQTT broker, enriches payloads, and routes events to Redpanda topics based on sensor type using Bloblang expressions.
3. A quality service consumes temperature and pressure events, runs SPC analysis (mean +/- 3 sigma), and publishes alerts when thresholds are breached.
4. A maintenance service consumes vibration events and applies threshold-based anomaly detection.
5. A dashboard service materializes a last-known-state view of every machine and serves it over an HTTP API.

Prerequisites

• Docker Desktop (includes Docker Engine and Docker Compose)
• Python 3.10 or later
• pip

Quick Start

Install Python dependencies:

pip3 install -r sensor_simulator/requirements.txt \
             -r quality_service/requirements.txt \
             -r maintenance_service/requirements.txt \
             -r dashboard_service/requirements.txt

Start the infrastructure (Redpanda, Mosquitto, Redpanda Connect, Console):

docker compose up -d

Wait a few seconds for healthchecks, then verify topics:

docker exec redpanda rpk topic list

Run each service in a separate terminal:

# Terminal 1: Sensor simulator
python3 sensor_simulator/simulator.py

# Terminal 2: Quality SPC service
python3 quality_service/service.py

# Terminal 3: Maintenance service
python3 maintenance_service/service.py

# Terminal 4: Dashboard service
python3 dashboard_service/service.py

Query the dashboard:

curl -s http://localhost:5050/machines | python3 -m json.tool
curl -s http://localhost:5050/alerts | python3 -m json.tool

Open Redpanda Console to inspect topics and messages visually.

Project Structure

event-driven-redpanda/
|-- docker-compose.yml          # Redpanda, Mosquitto, Connect, Console
|-- mosquitto.conf              # MQTT broker config
|-- connect.yaml                # Redpanda Connect pipeline (MQTT -> Redpanda)
|
|-- sensor_simulator/
|   |-- simulator.py            # Publishes fake MQTT telemetry for 3 machines
|   +-- requirements.txt
|
|-- quality_service/
|   |-- service.py              # SPC rolling-window breach detection
|   +-- requirements.txt
|
|-- maintenance_service/
|   |-- service.py              # Vibration threshold anomaly detection
|   +-- requirements.txt
|
+-- dashboard_service/
    |-- service.py              # Last-known-state HTTP API (Flask, port 5050)
    +-- requirements.txt

Exposed Ports

Service	Port
Redpanda Kafka API	19092
Redpanda Console	8080
Mosquitto MQTT	1883
Dashboard HTTP API	5050

Cleanup

Stop the Python services with Ctrl+C, then tear down Docker:

docker compose down -v