Scala for Data Science
The talk includes a notebook for Spark Notebook, which provides a notebook metaphor for interactive Spark development using Scala. If you aren't familiar with the idea of a notebook interface, think of it as an enhanced REPL that makes it easy to edit and run (or rerun) code, plot results, mix in markdown-based documentation, etc.
However, if you don't want to go to the trouble of installing and using Spark Notebook, there are Markdown and PDF versions of the same content in the
Use Existing Docker
A docker container exists with the Spark Notebook available with the current notebooks.
Pull it from docker hub:
docker pull datafellas/scala-for-data-science:1.0-spark2
docker run --rm -it --net=host -m 8g datafellas/scala-for-data-science:1.0-spark2 bash
Start the services
On Linux, go to http://localhost:9000.
On Mac/Win, you'll probably have to use the VM's IP/Name.
Otherwise, install Spark Notebook, version 0.6.3 or later. You can use either Scala 2.10 or 2.11. In the commands below, we'll assume the root directory of this installation is
/path/to/spark-notebook. Just use your real path instead. Due to a bug in library path handling, you must start Spark Notebook from this directory.
We'll also use
/path/to/scala-for-data-science as the path to your local clone of this Git repo. Again, substitute the real path...
There is one environment variable that you must define,
NOTEBOOKS_DIR. Run the following commands to define this variable and start Spark Notebook.
For Linux or OSX, use the following:
export NOTEBOOKS_DIR=/path/to/scala-for-data-science/notebooks cd /path/to/spark-notebook bin/spark-notebook
For Windows, use the following:
set NOTEBOOKS_DIR=c:\path\to\scala-for-data-science\notebooks cd \path\to\spark-notebook bin\spark-notebook
To evaluate all the cells in a notebook, use the Cell > Run All menu item. You can evaluate one cell at a time with the ▶︎ button on the toolbar, or use "shift+return". Both options run the currently-selected cell and advance to the next cell. Note that the notebook copy in the repo includes the output from a run.
Grab the slides for the rest of the presentation here.