Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Python clone of Spark, a MapReduce alike framework in Python
branch: master

This branch is 921 commits behind douban:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
dpark
examples
tests
tools
.gitignore
AUTHORS
LICENSE
README
TODO
setup.py

README

Dpark is a Python clone of Spark, MapReduce computing 
framework supporting regression computation.

Word count example wc.py:

 from dpark import DparkContext
 ctx = DparkContext()
 file = ctx.textFile("/tmp/words.txt")
 words = file.flatMap(lambda x:x.split()).map(lambda x:(x,1))
 wc = words.reduceByKey(lambda x,y:x+y).collectAsMap()
 print wc

This scripts can run locally or on Mesos cluster without
any modification, just with different command arguments:

$ python wc.py
$ python wc.py -m process
$ python wc.py -m mesos

See examples/ for more examples.

Some Chinese docs: https://github.com/jackfengji/test_pro/wiki
Something went wrong with that request. Please try again.