⚡ Distributed MapReduce System

Bring your own JAR, we'll bring the cluster

---

This is a cloud-native MapReduce platform, built from scratch, that runs on real Kubernetes, real OAuth2, and real S3-compatible storage. No Hadoop, no HDFS, no "trust me it scales" — you upload a JAR, the cluster spins up pods, and word count (or whatever you write) happens across machines instead of on one very tired laptop.

Built as part of the Principles of Distributed Systems course at the Technical University of Crete (TUC)

it works on my minikube, and honestly that's more than most distributed systems can say

---

Table of Contents

• The Elevator Pitch
• Architecture
• The Services
• How a Job Actually Runs
• Failure Handling
• Tech Stack
• Getting Started
• Writing Your Own Job
• Using the CLI
• Project Structure

---

🎯 The Elevator Pitch

#-the-elevator-pitch

The textbook MapReduce example is always word count: huge file, count every word, too slow for one machine, so you split the work across many. That's the whole idea here — except instead of faking it with threads on one box, this actually schedules real Kubernetes pods to do the work and cleans up after itself when they're done.

The fun part is how it's wired together:

• 🪣 MinIO instead of HDFS — a self-hosted S3, because rolling your own distributed filesystem for a course project felt like a bad life choice
• 📦 Workers aren't servers — each Map/Reduce task is a Kubernetes batch Job: a pod is born, does exactly one thing, and exits
• 🧩 You bring your own code — upload a JAR implementing Mapper/Reducer, and the worker finds it at runtime via Java's ServiceLoader, no recompiling the platform required
• 🔐 Keycloak handles auth — real OAuth2/OIDC, real JWTs, real JWKS validation, not a hardcoded if (password == "admin")
• 💻 A CLI called mr — with an interactive shell, because typing curl commands all day is nobody's idea of fun

---

🏗️ Architecture

#️-architecture

                         ┌─────────────┐
                         │   mr (CLI)  │   ← you, typing things
                         └──────┬──────┘
                                │ HTTPS + JWT
                                ▼
                      ┌───────────────────┐
                      │     ui-service     │  ← the front door
                      │  (auth, gateway)   │
                      └─────────┬──────────┘
                                │
                   ┌────────────┼────────────┐
                   ▼            ▼            ▼
             ┌──────────┐ ┌──────────┐ ┌────────────┐
             │ Keycloak │ │ manager- │ │  (nothing  │
             │  (auth)  │ │ service  │ │  else, it's│
             └──────────┘ │"the brain"│  the gateway)│
                          └─────┬─────┘ └────────────┘
                                │ creates & watches
                                ▼
                      ┌───────────────────┐
                      │  Kubernetes Jobs   │
                      │  (worker pods —    │
                      │   one per task)    │
                      └─────────┬──────────┘
                                │ read/write
                                ▼
                      ┌───────────────────┐
                      │  MinIO (S3-alike) │
                      └───────────────────┘
                                ▲
                                │ owns state
                      ┌───────────────────┐
                      │    PostgreSQL      │
                      │ (jobs, tasks, meta)│
                      └───────────────────┘

Five moving pieces, one namespace, zero patience for flaky infra.

---

🧱 The Services

#-the-services

Service	Role	The Vibe
ui-service	Public API gateway, validates JWTs, talks to Keycloak	The bouncer at the door
manager-service	Owns Postgres + MinIO + K8s, runs the whole job lifecycle	The brain — does all the actual thinking
worker	A CommandLineRunner that starts, does one task, reports back, and dies	Lives fast, dies young, leaves a part-0.txt behind
cli (mr)	Fat-JAR command line tool with an interactive shell	Your actual hands on the keyboard
common	Shared DTOs, enums, and the Mapper/Reducer interfaces	The glue nobody notices until it's missing

The Mapper / Reducer contract

Everything a user writes boils down to two interfaces:

public interface Mapper {
    void map(String inputKey, String line, BiConsumer<String, String> emit);
}

public interface Reducer {
    void reduce(String key, List<String> values, BiConsumer<String, String> emit);
}

emit is just a callback — instead of building and returning a list, you call emit.accept(key, value) whenever you've got something. The worker loads your implementation at runtime via ServiceLoader, so your JAR just needs a META-INF/services/...Mapper file pointing at your class. No platform recompile, no jar-in-a-jar gymnastics.

---

🔄 How a Job Actually Runs

#-how-a-job-actually-runs

1. You upload your data file and your code JAR — they land in MinIO under users/{you}/raw/... and users/{you}/code/..., with a row in Postgres tracking who owns what.
2. You submit the job — the manager creates a Job row (INITIALIZING), hands you back a jobId immediately, and does everything else asynchronously so your CLI isn't just sitting there.
3. Map phase kicks off — your input gets split into chunks, one Kubernetes batch/v1/Job spins up per chunk, each pod downloads your JAR, loads your Mapper via ServiceLoader, chews through its slice of the data, and writes intermediate results back to MinIO.
4. Reduce phase follows — once every map task reports COMPLETED, the manager gathers the intermediate output, splits it across your requested number of reducers, and launches another wave of pods running your Reducer.
5. You get your results — once every reduce task finishes, the manager generates presigned MinIO URLs (valid for an hour) so you can download the output straight from object storage, no API roundtrip needed.

The whole thing is driven by workers reporting their own status back over HTTP (IN_PROGRESS → COMPLETED/FAILED) plus a heartbeat every 10 seconds — so the manager always knows who's alive.

---

💀 Failure Handling (because things die)

#-failure-handling-because-things-die

Three layers of "please don't make me rerun the whole job":

Layer	What it catches	What happens
Kubernetes backoffLimit	Pod crashes before it can even talk to the manager	K8s itself retries the pod (up to 3 times)
Manager retry logic	Worker explicitly reports FAILED	Old K8s Job is deleted, a fresh one is launched for the same task — up to 3 attempts
Heartbeat watchdog	Worker goes silent (OOM, node died, network partition)	If no heartbeat in 30s, the task is declared dead and dropped into the retry logic above

If a reduce worker dies, only the reduce reruns — the map output is still safely sitting in MinIO. Nobody has to redo work that already succeeded.

---

🛠️ Tech Stack

#️-tech-stack

Piece	What it's doing here
Spring Boot 3.3.4	Powers ui-service, manager-service, and worker
Fabric8 Kubernetes Client	Lets the manager create/delete Jobs at runtime
MinIO	S3-compatible object storage for data, code, and results
Keycloak	OAuth2/OIDC — real JWTs, real JWKS validation
PostgreSQL + Flyway	Job/task/file state, versioned schema migrations
picocli	Powers the mr CLI, including its interactive shell mode
Micrometer + Actuator	Health probes and Prometheus metrics

---

🚀 Getting Started

#-getting-started

Prerequisites

• Java 21
• Docker (for Compose) or a local Kubernetes cluster (Minikube/Kind) for the full experience
• Maven

Local dev with Docker Compose

git clone https://github.com/kdani3/Distributed-MapReduce-System.git
cd Distributed-System
docker compose up -d

This brings up Postgres, MinIO, Keycloak, manager-service, and ui-service. Note: without a real Kubernetes cluster in the loop, worker pods can't actually be scheduled from Compose alone — point the manager at a local cluster (Minikube/Kind) if you want to run real jobs end-to-end.

Full Kubernetes deployment

kubectl apply -f k8s/namespace/
kubectl apply -f k8s/postgres/
kubectl apply -f k8s/minio/
kubectl apply -f k8s/keycloak/
kubectl apply -f k8s/manager/
kubectl apply -f k8s/ui/

Build the CLI

cd cli
./mr    # builds itself with Maven if mr.jar isn't there yet, then drops you into the shell

---

✍️ Writing Your Own Job

#️-writing-your-own-job

The bundled example is, unsurprisingly, word count:

public class WordCountMapper implements Mapper {
    @Override
    public void map(String inputKey, String line, BiConsumer<String, String> emit) {
        for (String token : line.split("\\s+")) {
            String word = token.toLowerCase().replaceAll("[^a-zA-Z0-9']", "");
            if (!word.isBlank()) emit.accept(word, "1");
        }
    }
}

public class WordCountReducer implements Reducer {
    @Override
    public void reduce(String key, List<String> values, BiConsumer<String, String> emit) {
        emit.accept(key, String.valueOf(values.stream().mapToLong(Long::parseLong).sum()));
    }
}

Register both in META-INF/services/, package it as a JAR, and you're ready to submit — no dependency on the platform's internals required.

---

💻 Using the CLI

#-using-the-cli

mr login                          # prompts for username/password, saves a JWT locally
mr run data.txt wordcount.jar -r 2   # upload + submit + watch, all in one go
mr status <jobId> --watch 5       # poll until it's done
mr results <jobId>                # download output, render a little bar chart in your terminal

Yes, the results command really does draw a bar chart out of █ characters. It's more satisfying than a raw CSV, fight us.

---

📁 Project Structure

#-project-structure

Distributed-System/
├── common/            — shared Mapper/Reducer interfaces, DTOs, enums
├── ui-service/         — public API gateway + auth
├── manager-service/    — orchestration brain: DB, MinIO, K8s
├── worker/             — the pod that does one task and exits
├── cli/                — the mr command-line tool
├── examples/wordcount/ — sample Mapper/Reducer implementation
├── k8s/                — all Kubernetes manifests
└── docker-compose.yml  — local dev environment

---

🙏 Special Thanks

To Kubernetes, for making "just restart it" an actual architecture pattern.

To whichever pod crashed the most times during testing — you taught us everything we know about backoffLimit.

Built with and pure anxiety for distributed systems failing in new and creative ways.