Learn the pyspark API through pictures and simple examples
Jupyter Notebook Shell
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

pyspark-pictures

Learn the pyspark API through pictures and simple examples

View on NBViewer

RDD Example:

example image

# flatMap
x = sc.parallelize([1,2,3])
y = x.flatMap(lambda x: (x, 100*x, x**2))
print(x.collect())
print(y.collect())

[1, 2, 3]
[1, 100, 1, 2, 200, 4, 3, 300, 9]

Install (for interactive use)

  1. install Spark
  2. install IPython notebook

Quick Start

  1. start pyspark inside IPython notebook

    IPYTHON_OPTS="notebook" pyspark
  2. open browser to notebook link

  3. open pyspark-pictures.ipynb or pyspark-pictures-dataframes.ipynb

  4. edit example code, press: ctrl + enter to run each cell

References

pyspark API

Contribute

Contributors are welcome
Original images are here, download to pdf, convert to svg with: genSVD (pdf2svg)