Data-Intensive Applications on HPC Using Hadoop, Spark and RADICAL-Cybertools (Supercomputing 2015 Tutorial)
Thank you for the great tutorial. We had a total of 63 participants! 31 participants successfully participated in the exercises running Hadoop, Spark and RADICAL-Pilot. See page below for archived materials!
- [Notebooks] (http://nbviewer.ipython.org/github/radical-cybertools/supercomputing2015-tutorial/tree/master/)
Links to Tool Documentations
This tutorial is based on a set of iPython notebooks, which allows us to interactively create and execute code from the web browser. If you are not familiar with iPython notebooks, we recommend to take some time to familiarize yourself with iPython:
- Documentation: http://ipython.readthedocs.org/en/stable/overview.html
The archived notebooks can be viewed with [NBViewer] (http://nbviewer.ipython.org/github/radical-cybertools/supercomputing2015-tutorial/tree/master/).
Exercises (Environment Not Active After Tutorial)
Please walk through the Examples located in our Jupyter notebooks:
Log in using your assigned tutorial user (it is the same user as your Linux/SSH user):
You will see an empty list folder in the browser:
Checkout the Tutorials Github repository by creating an empty notebook (Python2). You can execute bash commands using the
!before the command. Create a cell with the following content and execute it by pressing shift-enter:
!git clone https://github.com/radical-cybertools/supercomputing2015-tutorial.git
You will see a new folder
Go back to the overview screen and enter the
The tutorial overview notebooks gives an overview and links to the individual notebooks:
Please go to the Jupyterhub login page and execute step 1-3.
Log into edge node (also referred to as master node) with your tutorial user:
Test keyless login on Tutorial Edge Node:
ssh localhost /bin/date
This should work out of the box with the SSH key that has been automatically setup with your account. If it fails, you need to create a keyless login:
ssh-keygen -t rsa cd .ssh cat id_rsa.pub >> authorized_keys chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys
Verify that there is a notebooks directory
~/notebooksin your home folder!