Distributed Data Management System

This project is a robust, scalable distributed database system built with Node.js. It features sharding, replication, and a fault-tolerant recovery mechanism. The entire system is containerized with Docker for easy deployment and management.

Key Features

• Scalable Architecture: Uses sharding to partition data across multiple server nodes, enabling horizontal scaling.
• Consistent Hashing: Employs consistent hashing to balance the load of requests across available servers for each shard.
• Replication: Each data shard is replicated across multiple servers for high availability and read performance.
• Automatic Fault Tolerance: The load balancer continuously monitors server health. If a server fails, it is automatically removed, and a new replica is spawned with the correct shard data restored from other replicas.
• Containerized: Uses Docker and Docker Compose for easy, consistent deployment across different environments.
• RESTful API: Provides a simple API for initializing the database, managing servers, and performing data operations.

---

How to Run

Prerequisites

• Docker & Docker Compose
• Node.js & npm (for local development)

1. Setup (One-Time Only)

The load balancer container needs permission to communicate with the Docker daemon on your host machine to manage other containers.

For Linux or macOS: Open your terminal and run the following command to set permissions for the Docker socket.

sudo chmod 666 /var/run/docker.sock

For Windows: You must modify the docker-compose.yml file. Change the volumes section under the load_balancer service to mount the Windows named pipe:

# In docker-compose.yml under the load_balancer service:
volumes:
  - '//./pipe/docker_engine:/var/run/docker.sock'

Additionally, you may need to expose the Docker Daemon via TCP.

1. Open Docker Desktop Settings.
2. Go to the General tab.
3. Enable "Expose daemon on tcp://localhost:2375 without TLS".
4. Click "Apply & Restart".

Then, update load_balancer/src/loadBalancer.js to connect via TCP:

// Replace the Docker setup line with this:
const docker = new Docker({ host: 'host.docker.internal', port: 2375 });

2. Start the System

This command will build the Docker images and start all the containers in the background.

npm start

3. Interact with the System

Use the client/index.html file in your browser to interact with the API.

• Endpoint: http://localhost:5000
• Initialize the cluster: The first step is to send a POST request to /init to create the shards and spawn the initial set of servers. Use the "Init" preset in the client UI.

4. View Logs

To see the real-time logs from the load balancer and any running servers:

npm run logs

5. Stop the System

This command will stop and remove all running containers and the network.

npm stop