This project aims to provide a scalable and efficient MapReduce framework designed to run on Azure Kubernetes Service (AKS) leveraging the power of Go programming language, Docker containers, and gRPC for communication between nodes.
-
Submit a MapReduce job to the master node's API with input data location, mapper function, reducer function, and output data location.
-
The master node will distribute the input data across worker nodes, assign map tasks, collect intermediate results, and then assign reduce tasks.
-
Worker nodes will execute map and reduce tasks, storing intermediate results in Azure Storage.
-
Once all tasks are completed, the master node will aggregate the results and store them in the specified output location.