Skip to content
This repository has been archived by the owner on May 8, 2024. It is now read-only.

No-DFS and compression patches #3

Merged
merged 2 commits into from May 6, 2012
Merged

No-DFS and compression patches #3

merged 2 commits into from May 6, 2012

Commits on May 6, 2012

  1. Remove reliance on a distributed file system. 

    If DPARK_WORK_DIR is set instead of DPARK_SHARE_DIR, a web server will
be started on each slave to serve files to the other slaves.
    @Ssmithcr
    Ssmithcr committed May 6, 2012
    Copy the full SHA
    2c190a4 View commit details
    Browse the repository at this point in the history

  2. Use compression for the intermediate shuffle files. 

    I tried Google's Snappy compression, but it still left my nodes I/O
bound; I found zlib level 1 to be the best compromise between CPU and
I/O.  Compression in this case takes around the same amount of time
that marshalling takes; however decompression is much less than the
time to unmarshal.
    @Ssmithcr
    Ssmithcr committed May 6, 2012
    Copy the full SHA
    fc7c116 View commit details
    Browse the repository at this point in the history