Andre Luckow edited this page Nov 16, 2015 · 42 revisions

Data-Intensive Applications on HPC Using Hadoop, Spark and RADICAL-Cybertools (Supercomputing 2015 Tutorial)

Thank you for the great tutorial. We had a total of 63 participants! 31 participants successfully participated in the exercises running Hadoop, Spark and RADICAL-Pilot. See page below for archived materials!

Material

Links to Tool Documentations

Infrastructure

This tutorial is based on a set of iPython notebooks, which allows us to interactively create and execute code from the web browser. If you are not familiar with iPython notebooks, we recommend to take some time to familiarize yourself with iPython:

The archived notebooks can be viewed with [NBViewer] (http://nbviewer.ipython.org/github/radical-cybertools/supercomputing2015-tutorial/tree/master/).

Exercises (Environment Not Active After Tutorial)

Please walk through the Examples located in our Jupyter notebooks:

http://sc15-aws.radical-cybertools.org

  1. Log in using your assigned tutorial user (it is the same user as your Linux/SSH user):

    Login

    You will see an empty list folder in the browser:

    Login

  2. Checkout the Tutorials Github repository by creating an empty notebook (Python2). You can execute bash commands using the ! before the command. Create a cell with the following content and execute it by pressing shift-enter:

     !git clone https://github.com/radical-cybertools/supercomputing2015-tutorial.git
    

    Login

    You will see a new folder supercomputing2015-tutorial:

    Login

  3. Go back to the overview screen and enter the supercomputing2015-tutorial folder:

    Login

    The tutorial overview notebooks gives an overview and links to the individual notebooks:

    Login

Please go to the Jupyterhub login page and execute step 1-3.

Troubleshooting

  1. Log into edge node (also referred to as master node) with your tutorial user:

     ssh <user>@sc15-aws.radical-cybertools.org
    
  2. Test keyless login on Tutorial Edge Node:

     ssh localhost /bin/date
    

    This should work out of the box with the SSH key that has been automatically setup with your account. If it fails, you need to create a keyless login:

     ssh-keygen -t rsa
     cd .ssh
     cat id_rsa.pub >> authorized_keys
     chmod 700 ~/.ssh
     chmod 600 ~/.ssh/authorized_keys
    
  3. Verify that there is a notebooks directory ~/notebooks in your home folder!

Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.