Real-time Traffic Visualization using Kafka, Node.js, and Python

This project demonstrates real-time traffic visualization using Kafka for message queuing, Node.js for producing and consuming traffic data, and Python for visualizing the data using Matplotlib.

Description

The project consists of three main components:

• Kafka Cluster: Set up using Docker containers for message queuing.
• Node.js Producer and Consumer: Simulate and consume traffic data, respectively, using Kafka.
• Python Visualization: Real-time visualization of traffic data using Matplotlib.

Simulation of Real-Time Data

Real-time traffic data is simulated using random numbers generated by the Node.js producer script. These numbers represent the number of vehicles at different traffic signals in the city.

Steps to Run the Project

1. Start Zookeeper Container:

   docker run -d \
    --name zookeeper \
    -p 2181:2181 \
    confluentinc/cp-zookeeper:7.0.1
   

2. Start Confluent Kafka Container:

   docker run -d \
    --name kafka \
    -p 9092:9092 \
    --link zookeeper:zookeeper \
    -e KAFKA_BROKER_ID=1 \
    -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
    -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
    -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
    confluentinc/cp-kafka:7.0.1
   

3. Install Dependencies (for consumer and Python visualization):

   npm i kafkajs
   pip install -r requirements.txt
   

4. Run the Admin Script:

   node admin.js
   

5. Run the Producer:

   node producer.js
   

6. Run the Consumer (in a separate terminal):

   node consumer.js
   

7. Run the Python Visualization Script (in a separate terminal):

   python visualization.py
   

Note:

To simulate real-time traffic data, we have synthesized random numbers representing the number of vehicles at different traffic signals.