A simple MapReduce implementation written in Go, developed as part of an Advanced Distributed Systems class assignment. This project is based on an MIT assignment and demonstrates distributed processing using two implementations: one with RPC and another with gRPC.
This project implements a distributed MapReduce framework, showcasing the fundamentals of distributed computing. It processes large datasets by dividing tasks into map and reduce operations, distributed across multiple workers. The project includes two distinct implementations to explore both RPC and gRPC communication methods.
-
RPC Implementation:
A basic implementation of MapReduce using Go's built-in Remote Procedure Call (RPC) library. -
gRPC Implementation:
An extended version utilizing gRPC for more robust and efficient communication. -
Distributed Task Management:
Handles task distribution and fault tolerance across multiple workers.
The implementation files are located in the src/mr folder.
src/mr: Contains the core implementation of MapReduce using both RPC and gRPC.- Other supporting files and configurations are also included in the
srcdirectory.
To set up the project locally:
-
Clone the repository:
git clone https://github.com/mohaali482/distributed-map-reduce.git cd distributed-map-reduce/src -
Install dependencies:
Ensure you have Go installed on your system. Install required dependencies for gRPC:go mod tidy
-
Navigate to the project directory:
cd src/main -
Run the tests MapReduce implementation:
bash test-mr.sh
- Programming Language: Go
- Communication Protocols: RPC, gRPC
This project is licensed under the MIT License. See the LICENSE file for more details.