Skip to content
/ MapReduce Public

Scalable, fault tolerant implementation of Map Reduce in Go.

Notifications You must be signed in to change notification settings

a3y3/MapReduce

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

21 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MapReduce

This is an implementation of a highly scalable, fault tolerant and performant map reduce library.

  • Highly scalable: No hard limit on the number of "worker" nodes. Adding new worker nodes will scale the processing linerarly.
  • Fault tolerant: Support for detecting if worker nodes have gone down, then assigning any incomplete tasks to other workers.
  • Performant: Support for detecting straggler nodes, then assigning their tasks to other nodes.

Run

In the directory src/main run:

go run -race mrcoordinator.go data/pg-*

In another window, cd to the same directory and run:

go build -race -buildmode=plugin ../mrapps/wc.go && go run -race mrworker.go wc.so

This will produce an output of N files with the format mr-out-X. To view the complete output, run: cat mr-out-* | sort | more.

Test

Run test-mr.sh.

About

Scalable, fault tolerant implementation of Map Reduce in Go.

Topics

Resources

Stars

Watchers

Forks