Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Branch: master
Fetching contributors…

Cannot retrieve contributors at this time

50 lines (38 sloc) 1.29 KB

DPark is a Python clone of Spark, MapReduce(R) alike computing framework supporting regression computation.

Example for word counting (

 import dpark
 file = dpark.textFile("/tmp/words.txt")
 words = file.flatMap(lambda x:x.split()).map(lambda x:(x,1))
 wc = words.reduceByKey(lambda x,y:x+y).collectAsMap()
 print wc

This scripts can run locally or on Mesos cluster without any modification, just with different command arguments:

$ python
$ python -m process
$ python -m host[:port]

See examples/ for more examples.

Some Chinese docs:

DPark can run with Mesos (0.9 or latest).

If $MESOS_MASTER was configured, then you can run it with mesos just typing

$ python -m mesos

for shutcut. $MESOS_MASTER can be any scheme of mesos master, such as

$ export MESOS_MASTER=zk://zk1:2181,zk2:2181,zk3:2181/mesos_master 

In order to speed up shuffing, should deploy Nginx at port 5055 for accessing data in DPARK_WORK_DIR (default is /tmp/dpark), such as:

        server {
                listen 5055;
                server_name localhost;
                root /tmp/dpark/;

Mailing list: (

Jump to Line
Something went wrong with that request. Please try again.