Python clone of Spark, a MapReduce alike framework in Python
Switch branches/tags
Nothing to show
Pull request Compare This branch is 600 commits behind douban:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
dpark
examples
tests
tools
.gitignore
AUTHORS
CONTRIBUTORS
LICENSE
README.md
TODO
setup.py

README.md

DPark is a Python clone of Spark, MapReduce(R) alike computing framework supporting regression computation.

Example for word counting (wc.py):

 import dpark
 file = dpark.textFile("/tmp/words.txt")
 words = file.flatMap(lambda x:x.split()).map(lambda x:(x,1))
 wc = words.reduceByKey(lambda x,y:x+y).collectAsMap()
 print wc

This scripts can run locally or on Mesos cluster without any modification, just with different command arguments:

$ python wc.py
$ python wc.py -m process
$ python wc.py -m host[:port]

See examples/ for more examples.

Some Chinese docs: https://github.com/jackfengji/test_pro/wiki

DPark can run with Mesos (0.9 or latest).

If $MESOS_MASTER was configured, then you can run it with mesos just typing

$ python wc.py -m mesos

for shutcut. $MESOS_MASTER can be any scheme of mesos master, such as

$ export MESOS_MASTER=zk://zk1:2181,zk2:2181,zk3:2181/mesos_master 

In order to speed up shuffing, should deploy Nginx at port 5055 for accessing data in DPARK_WORK_DIR (default is /tmp/dpark), such as:

        server {
                listen 5055;
                server_name localhost;
                root /tmp/dpark/;
        }

Mailing list: dpark-users@googlegroups.com (http://groups.google.com/group/dpark-users)