Skip to content
Andre Luckow edited this page Nov 16, 2015 · 42 revisions

Data-Intensive Applications on HPC Using Hadoop, Spark and RADICAL-Cybertools (Supercomputing 2015 Tutorial)

Thank you for the great tutorial. We had a total of 63 participants! 31 participants successfully participated in the exercises running Hadoop, Spark and RADICAL-Pilot. See page below for archived materials!


Links to Tool Documentations


This tutorial is based on a set of iPython notebooks, which allows us to interactively create and execute code from the web browser. If you are not familiar with iPython notebooks, we recommend to take some time to familiarize yourself with iPython:

The archived notebooks can be viewed with [NBViewer] (

Exercises (Environment Not Active After Tutorial)

Please walk through the Examples located in our Jupyter notebooks:

  1. Log in using your assigned tutorial user (it is the same user as your Linux/SSH user):


    You will see an empty list folder in the browser:


  2. Checkout the Tutorials Github repository by creating an empty notebook (Python2). You can execute bash commands using the ! before the command. Create a cell with the following content and execute it by pressing shift-enter:

     !git clone


    You will see a new folder supercomputing2015-tutorial:


  3. Go back to the overview screen and enter the supercomputing2015-tutorial folder:


    The tutorial overview notebooks gives an overview and links to the individual notebooks:


Please go to the Jupyterhub login page and execute step 1-3.


  1. Log into edge node (also referred to as master node) with your tutorial user:

     ssh <user>
  2. Test keyless login on Tutorial Edge Node:

     ssh localhost /bin/date

    This should work out of the box with the SSH key that has been automatically setup with your account. If it fails, you need to create a keyless login:

     ssh-keygen -t rsa
     cd .ssh
     cat >> authorized_keys
     chmod 700 ~/.ssh
     chmod 600 ~/.ssh/authorized_keys
  3. Verify that there is a notebooks directory ~/notebooks in your home folder!

Clone this wiki locally
You can’t perform that action at this time.