ghostdsb / map_reduce Public

Notifications You must be signed in to change notification settings
Fork 0
Star 2

Distributed MapReduce model implemented in Elixir

2 stars 0 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
assets		assets
config		config
lib		lib
priv/resources		priv/resources
test		test
.formatter.exs		.formatter.exs
.gitignore		.gitignore
.tool-versions		.tool-versions
Makefile		Makefile
README.md		README.md
mix.exs		mix.exs
mr-out-0		mr-out-0

Repository files navigation

Distributed MapReduce

MapReduce model implemented as per this paper

MapReduce: Simplified Data Processing on Large Clusters

Overview

Execution

Master

keeps track of map jobs, reduce jobs and backlog jobs
pings worker at regular intervals about job status
in case worker is down, pushes its job to backlog
when worker asks for job, assigns one from backlog, map, or reduce jobs

Worker

connects to master
asks for job
pings master when job is done

In Action

Usage

start a master: make master
start workers: make worker NAME=worker_name

About

Distributed MapReduce model implemented in Elixir

distributed-systems elixir

Report repository

Languages