sparklyr_test

Testing the sparklyr package for data wrangling & basic modelling.

https://aurora-mareviv.github.io/sparklyr_test

Contents

Jupyter Notebooks

sparklyr_test is a Jupyter notebook for interactive data exploration & modelling.

sparklyr_test2 is another Jupyter notebook for interactive data exploration & modelling with a -much- bigger dataset. Visualize it in nbviewer.jupyter.org

R script

sparklyr_script is a .R script that launches as an application.

$ spark-submit --deploy-mode client sparklyr_script.R

Clone this repo

$ git clone https://github.com/aurora-mareviv/sparklyr_test

Tutorial

Materials to start using Spark in R (sparklyr package).

• My slides for the sparklyr workshop - updated July 2019.
  • In Spanish: sparklyr workshop
• Examples for local usage (in RStudio): sparklyr_start.Rmd.
  • Rendered in HTML sparklyr_start.html
• Examples for cluster usage (in Jupyter): sparklyr_test.ipynb.
  • Rendered in nbviewer.jupyter.org

References

Sparklyr

• sparklyr workshop
• sparklyr presentation by RStudio
• sparklyr tutorial.
• sparklyr cheatsheet.
• sparklyr: creating extensions.
• Differences between sparkr and sparklyr.
• Hive Operators and UDFs.
• String Functions in Hive.
• POSIX regular expressions.

PySpark

• PySpark course
• Material for Machine Learning Workshop Galicia 2016.
• PySpark Programming Guide.
• PySpark cheatsheet.

Jupyter

• Jupyter shortcuts