Skip to content

youngkin/erlang-mapreduce

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

What?

A simple parallel MapReduce implementation in Erlang. The emphasis is on simplicity and understandability of the code.

Check out the demo files for simple examples, including:

  • building an inverted index of a collection of text documents
  • word frequency count of a collection of text documents
  • grep tool to search a collection of text documents

To compile:

make code

To run the inverted index demo:

cd ebin
erl
% in the erl prompt:
> Index = demo_inverted_index:index(test). % index the test subdirectory
> demo_inverted_index:query_index(Index, rover). % what files contain the word 'rover'?
{ok,["test/dogs","test/cars"]}
> halt().

To clean up:

make clean

Why?

This implementation is part of the teaching material of my course on multicore programming, a course I teach at the Vrije Universiteit Brussel (VUB) in Brussels, Belgium.

The goal is to teach students both the fundamentals of MapReduce (in particular, the API of the Map and Reduce operations, and how these are combined to formulate large data processing jobs), and to increase their fluency of Erlang at the same time. The code showcases process spawning, synchronization via message passing and process termination.

Slideware

I gave a 40-minute talk about this project at the Erlang Factory Lite Brussels. The slides are available here (pdf).

In Scala

I contributed a chapter discussing an adaptation of this MapReduce implementation in Scala in Philipp Haller's book on Actors in Scala (chapter 9, Distributed and Parallel Computing).

Acknowledgements

The inverted index example was taken from Joe Armstrong's Programming Erlang book.

Feedback

I welcome any feedback at tvcutsem at vub.ac.be. Or drop me a line on twitter.

About

Simple MapReduce implementation in Erlang

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Erlang 100.0%