RR-Team-83-yadtq-yet-another-distributed-task-queue-

A Distributed Task Queue is a system that helps manage tasks across multiple computers instead of relying on just one. This project enables efficient and reliable task distribution, ensuring seamless task execution even under high load or when individual systems are unavailable.

Features

• Manage tasks across multiple workers.
• Supports task queuing, distributed execution, and scaling.
• Integration with Kafka and Redis for reliable task processing.

---

Table of Contents

1. Installation
   • Kafka Installation
   • Redis Installation
   • Install Dependencies
2. Usage
   • Starting Kafka
   • Starting and Stopping Redis
   • Kafka Commands

---

Installation

Kafka Installation

For macOS/Linux:

1. Download Kafka from the official site.
2. Extract the downloaded file:

   tar -xzf kafka_2.13-3.8.1.tgz

3.  cd kafka_2.13-3.8.1
   

Redis Installation

For macOS:

1. Install Homebrew if not already installed:

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   

2. Install Redis:

   brew install redis
   

For Linux:

1. Update the package list:

   sudo apt update

2. Install Redis:

   sudo apt install redis
   

Install Dependencies:

1. Ensure you have Python and pip installed, then install the project dependencies using:

   pip install -r requirements.txt
   

Usage:

Starting Kafka

1. Navigate to the Kafka directory:

   cd kafka_2.13-3.8.1/

2. Start the Kafka server:

   bin/kafka-server-start.sh config/kraft/server.properties
   

Starting and stopping Redis

For MacOS:

1. Start Redis:

   brew services start redis

2. Stop Redis:

   brew services stop redis
   

For Linux:

1. Start Redis:

   sudo service redis-server start

2. Stop Redis:

   sudo service redis-server stop
   

Kafka Commands:

1. Changing the Number of Partitions for a Topic:

   ./kafka-topics.sh --alter --topic tasks --partitions 2 --bootstrap-server 127.0.0.1:9092

2. Adding a New Topic:

   ./kafka-topics.sh --create --topic tasks_add --bootstrap-server 127.0.0.1:9092 --partitions 3 --replication-factor 1
   ./kafka-topics.sh --create --topic tasks_sub --bootstrap-server 127.0.0.1:9092 --partitions 3 --replication-factor 1
   ./kafka-topics.sh --create --topic tasks_mul --bootstrap-server 127.0.0.1:9092 --partitions 3 --replication-factor 1

Add each topic separately, we have given the command for the task subtraction. Use the syntax task_task_name for including tasks