Visualizer

Visualizer is a project that showcases the power of real-time data visualization on a map. It uses D3.js, Kafka, Node.js, and MongoDB to create an interactive and dynamic dashboard. For demonstration purposes, it only tracks the last 100 (earthquakes with magnitude > 4.0) around the world and displays them on the map with relevant information. However, the application may be extended to include other types of data, such as weather, traffic, or pollution.

Table of Contents

• Introduction
• Technologies
• Installation
• Usage
• Code of Conduct
• Notice
• Contributing
• License

Introduction

Real-time data visualization is a useful and engaging way to explore and understand complex phenomena. It can reveal patterns, trends, and anomalies that are otherwise hidden or difficult to grasp. Visualizer is a project that leverages various technologies to create a real-time data visualization dashboard on a map. It scrapes data from different sources, stores them in a MongoDB database, processes them with Kafka, and serves them with Node.js and Express.js. It also uses Socket.io to enable real-time communication between the server and the client. On the front-end, it uses EJS to render HTML templates and D3.js to create interactive and responsive map visualizations. The application can handle different types of data, but for simplicity, it focuses on earthquake data from around the world. It shows the location, magnitude, depth, and time of the last 100 earthquakes on a map, and allows the user to filter and zoom in on the data. The application also updates the data every 10 minutes to reflect the latest events.

Visualizer consists of four main components:

• An Apache Kafka server that is used for handle and pass realtime data. between the application components.
• A producer that scrapes data from a website and saves them to a MongoDB database. It also periodically checks the database for new records and sends them through Kafka.
• A consumer that receives the messages through Kafka and highlights them on the map in real-time.
• A docker-compose file that orchestrates the deployment of all the components using Docker.

Visualizer allows users to see the location, magnitude, depth, and time of the most recent earthquakes on a map. Users can also zoom in and out, pan around, and click on each earthquake marker to see more details.

Visualizer is a fun and educational project that demonstrates how to use various technologies to create a data-driven web application.

Technologies

Visualizer uses the following technologies:

• Docker - A platform for building, running, and sharing applications using containers.
• D3.js - A library for manipulating documents based on data using HTML, SVG, and CSS.
• MongoDB - A document-based database that stores data in JSON-like format.
• Kafka - A distributed streaming platform that enables data pipelines and event-driven applications.
• Webpack - A module bundler that compiles and optimizes web assets.
• Zookeeper - A centralized service that maintains configuration information and provides coordination for distributed systems.
• Node.js - A JavaScript runtime environment that executes code outside the web browser.
• Express.js - A web framework for Node.js that simplifies web development.
• Socket.io - A library that enables real-time communication between web clients and servers.
• Mongoose - An object data modeling (ODM) library that simplifies working with MongoDB.
• Axios - A promise-based HTTP client that supports multiple browsers and Node.js.
• Cheerio - A fast and flexible library that parses HTML and XML documents.

Installation

To install and run Visualizer, you need to have Docker and Docker Compose installed on your machine. You just need to copy/download the docker-compose.yml file from the repository to a directory and then run the following command to start all the components:

docker-compose up

This will pull the required images from Docker Hub and build the producer and consumer images. It will also create a network and a volume for the containers to communicate and persist data.

Usage

Once all the components are up and running, you can access the application by opening your web browser and visiting http://localhost:3000 . You should see something like this:

This page shows the map with the earthquake markers. You can zoom in and out, pan around, and click on each marker to see more details. You can also see the status of the consumer and the number of messages received.

There is also a page at http://localhost:29009 that shows the last 10 records that were scraped, sorted with the newwest first.

You can customize some options for the producer and the consumer by using environment variables.

These are the available variables for the producer:

Variable	Description	Default
MONGO_URI	The database connection string.	mongodb://localhost:27017/earthquakes
DB_NAME	The database name.	earthquakes
DB_USER	The database username.	Optional
DB_PASSWORD	The database password.	Optional
WEB_SERVER_PORT	The web server port.	29009
KAFKA_BROKER	The Kafka broker address.	localhost:9092
KAFKA_TOPIC	The Kafka topic name.	earthquake
KAFKA_SEND_INTERVAL	The time cycle in milliseconds in which the producer will check for new records to send in the database.	10000
SCRAPER_INTERVAL	The time cycle in milliseconds in which the scraper will scrape the website.	600000 (10 minutes)

These are the available variables for the consumer:

Variable	Description	Default
PORT	The port number for the server to serve the page.	3000
KAFKA_BROKER	The address of the Kafka broker.	localhost:9092
KAFKA_TOPIC	The name of the Kafka topic.	earthquake

You can pass them as arguments when running docker-compose up. For example:

MONGO_URI=mongodb://user:pass@host:port/db

Code of Conduct

See the CODE OF CONDUCT file for more details.

Notice

See the NOTICE file for more details.

Contributing

See the CONTRIBUTING file for more details.

License

Visualizer is licensed under the GNU General Public License v3.0. See the LICENSE file for more details.