IoT Data Processing Pipeline

A multi-service IoT data processing system built with Scala, Kafka, and Docker.

Architecture

This project consists of several microservices:

1. iot-simulator - Generates realistic IoT sensor events (temperature, humidity, pressure)
2. iot-storage - Consumes IoT events and stores them in Kafka topics for analytics
3. alert-handler - Processes alerts and sends notifications for critical conditions
4. alert-selector - Filters specific alerts based on conditions (to be implemented)
5. service-metrics - Collects data from Kafka and stores in MinIO S3 (to be implemented)
6. service-analytics - Loads S3 data and exposes via Grafana (to be implemented)
7. spark-analyzer - Advanced analytics with Spark (to be implemented)

Quick Start

Prerequisites

• Docker and Docker Compose
• SBT (for local development)

Running the System

1. Start all services:

docker-compose up --build

2. Monitor the logs to see:

   • IoT events being generated by the simulator
   • Events being processed and stored by iot-storage
   • Alerts being handled by alert-handler

3. Access Kafka UI at http://localhost:8080 to monitor topics and messages

Kafka Topics

The system creates and uses these Kafka topics:

• iot-events - Raw IoT sensor data from simulator
• iot-storage - Processed events for storage
• iot-analytics - Enriched events for analytics
• filtered-alerts - Filtered alerts for notification

IoT Event Format

{
  "deviceId": "sensor-001",
  "timestamp": 1672531200000,
  "temperature": 23.5,
  "humidity": 45.2,
  "pressure": 1013.25,
  "location": {
    "latitude": 48.8566,
    "longitude": 2.3522
  },
  "status": "active"
}

Alert Conditions

Alerts are generated for:

• Temperature: > 40°C (WARNING), > 45°C (CRITICAL), < 0°C (WARNING), < -10°C (CRITICAL)
• Humidity: > 80% (WARNING), > 90% (CRITICAL), < 20% (WARNING)
• Pressure: < 980 hPa (WARNING), < 970 hPa (CRITICAL), > 1030 hPa (WARNING)
• Device Status: "critical" status generates CRITICAL alert

Development

Building Individual Services

# Build specific service
cd iot-simulator
sbt compile

# Run specific service locally (requires Kafka running)
sbt run

Testing

# Compile all services
sbt compile

# Test specific service
cd alert-handler
sbt test

Next Steps

1. Implement alert-selector service for advanced filtering
2. Add MinIO S3 storage integration in service-metrics
3. Create Grafana dashboards in service-analytics
4. Add Spark analytics in spark-analyzer
5. Add proper error handling and monitoring
6. Implement authentication and security