pmundkur / odisco

ODisco is a library for implementing Disco map-reduce jobs in OCaml.
It implements the Disco worker protocol, and provides the plumbing to
connect user-specified computations to Disco data sources and results.

Prerequisites:
--------------

The library has the following dependencies:

. ocaml 3.12.0 or later

. ocaml-findlib (Debian/Ubuntu/Fedora) / godi-findlib (Godi) / findlib (Gentoo, source)

. libzip-ocaml-dev (Debian/Ubuntu) / ocaml-zip (Fedora) / godi-zip (Godi) / camlzip (Gentoo, source)

. Disco 0.4 or later (http://discoproject.org)

. sonet

  This is a simple asynchronous networking library for OCaml.

  $ git clone git://github.com/pmundkur/sonet.git
  $ cd sonet && make && make install

  To uninstall:
  $ cd sonet && make uninstall

The tests have the following dependencies:

. atd    (https://github.com/MyLifeLabs/atd) / libatd-ocaml-dev (Debian/Ubuntu) / godi-atd (Godi)
. atdgen (https://github.com/MyLifeLabs/atdgen) / libatdgen-ocaml-dev (Debian/Ubuntu) / godi-atdgen (Godi)
. biniou (https://github.com/mjambon/biniou) / libbiniou-ocaml-dev (Debian/Ubuntu) / godi-biniou (Godi)

Building:
---------

 $ make

Installing:
-----------

 $ make install

Uninstalling:
-------------

 $ make uninstall

Using:
------

. Write a module implementing the lib/task.ml:TASK interface.  Pass
  this module to Worker.start in your main program.  See
  tests/test_install_task.ml for an example.

  Read the documentation in lib/task.mli and lib/worker.mli for more
  details.

. Build it and link to the ODisco library.  See tests/Makefile for an
  example build.

. Submit your compiled executable to Disco using 'disco job'.

  For example, to do a word count on a large text file:

  $ make -C tests
  $ wget -O /tmp/bigfile.txt http://discoproject.org/media/text/bigfile.txt
  $ ddfs push text /tmp/bigfile.txt
  $ ddfs blobs text \
    | disco job -m -r -p ocamlwc tests/word_count.opt \
    | xargs disco wait \
    | xargs ddfs cat