docs/lecture11_Odyssey.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 11: Odyssey!!!\n",
    "\n",
    "Date: 10/05/2017, Thursday"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are expected to finish 8+1 tiny tasks. They will help you get prepared for the final project!\n",
    "\n",
    "Related resources:\n",
    "\n",
    "- [Ryans's Linux tutorial](https://ryanstutorials.net/linuxtutorial/)\n",
    "- *Intro-to-Odssey-S17_am111.pdf* on Canvas.\n",
    "- [Odyssey quickstart guide](https://www.rc.fas.harvard.edu/resources/odyssey-quickstart-guide/)\n",
    "- [MATLAB on Odyssey](https://www.rc.fas.harvard.edu/resources/documentation/software/matlab/)\n",
    "- [Parallel MATLAB on Odyssey](https://www.rc.fas.harvard.edu/resources/documentation/software/parallel-matlab-pct-dcs/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1: Command line on your laptop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read [Session 4 note](./session4_linux.ipynb), especially [Ryans's tutorial](https://ryanstutorials.net/linuxtutorial/) if you didn't come to Monday's session. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After reading \n",
    "[Chapter 1](https://ryanstutorials.net/linuxtutorial/commandline.php) \n",
    "to [Chapter 5](https://ryanstutorials.net/linuxtutorial/filemanipulation.php), \n",
    "you should at least know the following Linux commands\n",
    "\n",
    "- `ls`\n",
    "- `pwd`\n",
    "- `mkdir`\n",
    "- `cd`\n",
    "- `mv`\n",
    "- `rm` and `rm -rf`\n",
    "- `cp` and `cp -r`\n",
    "\n",
    "If you choose *vi/vim* as your text editor, read [Chapter 6](https://ryanstutorials.net/linuxtutorial/vi.php). Then you should at least know the following *vim* commands\n",
    "\n",
    "- `i`\n",
    "- `esc`\n",
    "- `:wq`\n",
    "- `:q!`\n",
    "\n",
    "Find your own tutorial if you choose other text editors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing code in terminal\n",
    "\n",
    "**Task**: Use *vim* or other command line text editer to create a matlab file *hello.m* with the content \"disp('hello world!')\"\n",
    "\n",
    "We use *vim* as an example.\n",
    "\n",
    "First, create a text file by\n",
    "\n",
    "    vim hello.m\n",
    "\n",
    "(If *hello.m* already exists, then it will just open that file)\n",
    "\n",
    "Inside *vim*, type `i` to enter the *Insert Mode*.\n",
    "\n",
    "Then type the code as usual. For example\n",
    "\n",
    "    disp('hello world!')\n",
    "\n",
    "After writting the content, type `esc` to go back to *Command Mode*.\n",
    "\n",
    "Finally, type `:wq` to save and quit *vim*.\n",
    "\n",
    "Again, read [Chapter 6](https://ryanstutorials.net/linuxtutorial/vi.php) for more *vim* usages!\n",
    "\n",
    "\n",
    "**Tips**: You can check the content of *hello.m* by a graphic editer. \n",
    "On Mac, you can use `open ./` to open the graphic finder, \n",
    "and then open *hello.m* that you've just created. On Odyssey (See Task 2), there's no graphic editor, \n",
    "so you will also use *vim* to check the file content."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running MATLAB interactively in terminal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Windows users can jump to Task 2 because I am not sure if the following stuff would work.**\n",
    "\n",
    "Find the MATLAB executable path on your laptop. On Mac it should be something like\n",
    "\n",
    "    /Applications/MATLAB_R2017a.app/bin/matlab\n",
    "\n",
    "Running the above command will open the traditional graphic version of MATLAB. \n",
    "\n",
    "To only use the command line, add 3 options:\n",
    "\n",
    "    /Applications/MATLAB_R2017a.app/bin/matlab -nojvm -nosplash -nodesktop\n",
    "    \n",
    "Play with this command line version of MATLAB for a while. Type `exit` to quit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set shortcut\n",
    "If you are tired with typing this long command, you can set\n",
    "    \n",
    "    alias matlab='/Applications/MATLAB_R2017a.app/bin/matlab'\n",
    "    \n",
    "Then you can simply type `matlab` to launch the program. However, this shortcut will go away if you close the terminal. To make it a permanent configuration, add the above command to a system file called *~/.bash_profile*. You can edit it by *vim* for example:\n",
    "\n",
    "    vim ~/.bash_profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running MATLAB scripts in terminal\n",
    "\n",
    " `cd` to the directory where you saved the *hello.m* file. You can execute it by\n",
    " \n",
    "    matlab  -nojvm -nosplash -nodesktop\n",
    "    hello\n",
    " \n",
    " \n",
    "Or you can use '-r' to combine two commands together\n",
    "\n",
    "    matlab  -nojvm -nosplash -nodesktop -r hello\n",
    "    \n",
    "If you didn't set shortcut, the full command would be\n",
    "    \n",
    "    /Applications/MATLAB_R2017a.app/bin/matlab -nojvm -nosplash -nodesktop -r hello\n",
    "    \n",
    "(I actually prefer this command line version to the complicated graphic version!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 2: Command line on Odyssey"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Login"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Login to Odyssey by\n",
    "\n",
    "    ssh am111uXXXX@login.rc.fas.harvard.edu\n",
    "    \n",
    "Check [Odyssey website](https://www.rc.fas.harvard.edu/resources/access-and-login/) if you have any trouble.\n",
    "\n",
    "**Tips**: You can open multiple terminals and login to Odyssey, if one is not enough for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic navigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Repeat the basic Linux commands**, but this time on Odyssey, not on your laptop.\n",
    "\n",
    "You should see Mac and Linux (Odyssey) commands are almost identical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### File transfer\n",
    "\n",
    "#### Use scp\n",
    "\n",
    "You can transfer files by the built-in `scp` (security-copy) command. **Make sure you are running this command on your laptop, not on odyssey.**\n",
    "\n",
    "From you laptop to Odyssey (first figure out your Odyssey home directory path by `pwd`)\n",
    "\n",
    "    scp local_file_path username@login.rc.fas.harvard.edu:/path_shown_by_pwd_on_Odyssey\n",
    "\n",
    "**Try to transfer *hello.m* that you wrote in Task 1 to Odyssey!** You will be asked to enter your password again.\n",
    "\n",
    "From to Odyssey to your laptop is just reversing the arguments\n",
    "\n",
    "    scp username@login.rc.fas.harvard.edu:/file_path_on_odyssey local_file_path\n",
    "\n",
    "Use `scp -r` for transfering directory (similar to `cp -r`)\n",
    "\n",
    "#### Use other tools\n",
    "\n",
    "Use [Filezilla](https://www.rc.fas.harvard.edu/resources/documentation/sftp-file-transfer/)\n",
    "if you need to transfer a lot of file!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 3: MATLAB on Odyssey"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load MATLAB\n",
    "\n",
    "Load MATLAB by\n",
    "\n",
    "    module load matlab\n",
    "    \n",
    "(If you get an error, run `source new-modules.sh` and try again.)\n",
    "    \n",
    "It loads the lastest version by default. You can check the version by `which`\n",
    "    \n",
    "    [username]$ which matlab\n",
    "    alias matlab='matlab -singleCompThread'\n",
    "\t/n/sw/matlab-R2017a/bin/matlab\n",
    "    \n",
    "Or you can load a specific version\n",
    "    \n",
    "    module load matlab/R2017a-fasrc01\n",
    "    \n",
    "\n",
    "Use this [RC portal](https://portal.rc.fas.harvard.edu/apps/modules) to find avaiable software and the corresponding loading command. Search for MATLAB. How many different verions do you see?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run MATLAB\n",
    "\n",
    "After loading MATLAB, you can run it by: (same as on your laptop)\n",
    "\n",
    "    matlab -nojvm -nosplash -nodesktop\n",
    "    \n",
    "The 3 options are crucial because there's no graphical user interface on Odyssey.\n",
    "\n",
    "Play with it, and type `exit` to quit. \n",
    "\n",
    "Run *hello.m* by `matlab -nojvm -nosplash -nodesktop -r hello`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 4: Interactive Job on Odyssey"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After logging into Odyssey, you are on a *home node* with very few computational resources. For any serious computing work you need to switch to a *compute node*. The easiest way is to do this interactively ([more about interative mode](https://www.rc.fas.harvard.edu/resources/running-jobs/#Interactive_jobs_and_srun)):\n",
    "\n",
    "    srun -t 0-0:30 -c 4 -N 1 --pty -p interact /bin/bash\n",
    "\n",
    "Here we request 30 minutes of computing time (`-t 0-0:30`) on 4 CPUs (`-c 4`), on a single computer (`-N 1`), using interactive mode (`--pty` and `/bin/bash`).\n",
    "\n",
    "**Warning: Don't request too many CPUs! This will make you wait for much longer.**\n",
    "\n",
    "`-p interact` only means you are requesting CPUs on the *interactive partition*, but doesn't mean that you want it to run interactively. The following command starts interactive mode on the *general partition* ([more about partition](https://www.rc.fas.harvard.edu/resources/running-jobs/#SLURM_partitions)).\n",
    "\n",
    "    srun -t 0-0:30 -c 4 -N 1 --pty -p general /bin/bash\n",
    "\n",
    "**Then repeat what you've done in Task 3.**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 5: Batch Job on Odyssey"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your job runs for hours or even days, you can submit it as a *batch job*, so you don't need to keep your terminal open all the time. You are allowed to log out and go away while the job is runnning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a file called *runscript.sh* with the following content. (you can use *vim* to create such a text file)\n",
    "\n",
    "    #!/bin/bash\n",
    "    #SBATCH -J Matlabjob1\n",
    "    #SBATCH -p general\n",
    "    #SBATCH -c 1 # single CPU\n",
    "    #SBATCH -t 00:05:00\n",
    "    #SBATCH --mem=400M # memory \n",
    "    #SBATCH -o %j.o # output filename \n",
    "    #SBATCH -e %j.e # error filename\n",
    "    \n",
    "    ## LOAD SOFTWARE ENV ## \n",
    "    source new-modules.sh\n",
    "    module purge\n",
    "    module load matlab/R2017a-fasrc01\n",
    "\n",
    "    ## EXECUTE CODE ##\n",
    "    matlab -nojvm -nodisplay -nosplash -r hello\n",
    "\n",
    "It just puts the options you've used in Task 4 into a text file.\n",
    "\n",
    "Make sure *runscript.sh* is at the same directory as hello.m, then execute\n",
    "    \n",
    "    sbatch runscript.sh\n",
    "    \n",
    "Use `sacct` to check job status. You should get some output files once it is finished. (more about [submitting](https://www.rc.fas.harvard.edu/resources/running-jobs/#Submitting_batch_jobs_using_the_sbatch_command) and [monitoring](https://www.rc.fas.harvard.edu/resources/running-jobs/#Summary_of_SLURM_commands) jobs)\n",
    "\n",
    "**Tips: always test your code in interactive mode before submitting a batch job!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 6: Use MATLAB-parallel on your laptop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure you've installed the parallel toolbox. To start the command line version, remove the `-nojvm` option when using parallel mode. (The original graphic version works as usual)\n",
    "\n",
    "    matlab -nosplash -nodesktop\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize parallel mode by"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting parallel pool (parpool) using the 'local' profile ...\n",
      "connected to 2 workers.\n",
      "\n",
      "ans = \n",
      "\n",
      " Pool with properties: \n",
      "\n",
      "            Connected: true\n",
      "           NumWorkers: 2\n",
      "              Cluster: local\n",
      "        AttachedFiles: {}\n",
      "          IdleTimeout: 30 minutes (30 minutes remaining)\n",
      "          SpmdEnabled: true\n",
      "\n"
     ]
    }
   ],
   "source": [
    "parpool('local', 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then run this script for several times to make sure you get speed-up by using parallel for-loop (`parfor`) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serial time: 2.724932; result: 1000000000 \n",
      "parallel time: 1.748450; result: 1000000000 \n"
     ]
    }
   ],
   "source": [
    "n = 1e9;\n",
    "\n",
    "X = 0;\n",
    "tic\n",
    "for i = 1:n\n",
    "    X = X + 1;\n",
    "end\n",
    "T = toc;\n",
    "fprintf('serial time: %f; result: %d \\n',T,X)\n",
    "\n",
    "X = 0;\n",
    "tic\n",
    "parfor i = 1:n\n",
    "    X = X + 1;\n",
    "end\n",
    "T = toc;\n",
    "fprintf('parallel time: %f; result: %d \\n',T,X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tips**: For command line version of MATLAB, save the code as *parallel_timing.m*, and then execute `parallel_timing` inside MATLAB."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, quit the parallel mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "delete(gcp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 7: Use MATLAB-parallel on Odyssey interactive mode\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat what you've done in Task 6, but on Odyssey. This might **not** be as straightforward as you expected!\n",
    "\n",
    "\n",
    "You need to request enough memory for the parallel tool box\n",
    "\n",
    "    srun -t 0-0:30 -c 4 -N 1 --mem-per-cpu 4000 --pty -p interact /bin/bash\n",
    "\n",
    "Environment variable *SLURM_CPUS_PER_TASK* tells you how many CPUs are available\n",
    "\n",
    "    echo $SLURM_CPUS_PER_TASK\n",
    "    4\n",
    "\n",
    "For parallel support, you need to call `matlab-default` instead of `matlab` to launch the program, as described [here](https://www.rc.fas.harvard.edu/resources/documentation/software/parallel-matlab-pct-dcs/).\n",
    "\n",
    "    module load matlab\n",
    "    matlab-default -nosplash -nodesktop\n",
    "    \n",
    "Inside MATLAB, you can again check the number of CPUs by\n",
    "\n",
    "    getenv('SLURM_CPUS_PER_TASK')\n",
    "    ans = '4'\n",
    "    \n",
    "Initialize parallel mode by (this is a general code for any number of CPUs)\n",
    "\n",
    "    parpool('local', str2num(getenv('SLURM_CPUS_PER_TASK')) )\n",
    "    \n",
    "The initialization might take severals minutes on Odyssey. Eventually you should see something like\n",
    "\n",
    "    ans =\n",
    "\n",
    "     Pool with properties:\n",
    "\n",
    "                Connected: true\n",
    "               NumWorkers: 4\n",
    "                  Cluster: local\n",
    "            AttachedFiles: {}\n",
    "              IdleTimeout: 30 minutes (30 minutes remaining)\n",
    "              SpmdEnabled: true\n",
    "\n",
    "Then, execute the *parallel_timing.m* script in Task 6. You should see a speed-up like that\n",
    "\n",
    "    >> parallel_timing\n",
    "    serial time: 12.228084; result: 1000000000\n",
    "    parallel time: 2.667366; result: 1000000000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 8: MATLAB-parallel as batch Job"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sightly modify the script *parallel_timing.m* in Task 6. Call it *parallel_timing_batch.m* this time.\n",
    "\n",
    "    parpool('local', str2num(getenv('SLURM_CPUS_PER_TASK')))\n",
    "\n",
    "    n = 1e9;\n",
    "\n",
    "    X = 0;\n",
    "    tic\n",
    "    for i = 1:n\n",
    "        X = X + 1;\n",
    "    end\n",
    "    T = toc;\n",
    "    fprintf('serial time: %f; result: %d \\n',T,X)\n",
    "\n",
    "    X = 0;\n",
    "    tic\n",
    "    parfor i = 1:n\n",
    "        X = X + 1;\n",
    "    end\n",
    "    T = toc;\n",
    "    fprintf('parallel time: %f; result: %d \\n',T,X)\n",
    "    \n",
    "    X = 0;\n",
    "    tic\n",
    "    parfor i = 1:n\n",
    "        X = X + 1;\n",
    "    end\n",
    "    T = toc;\n",
    "    fprintf('parallel time: %f; result: %d \\n',T,X)\n",
    "    \n",
    "    delete(gcp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, change the *runscript.sh* in Task 5 correspondingly\n",
    "\n",
    "    #!/bin/bash\n",
    "    #SBATCH -J timing\n",
    "    #SBATCH -o timing.out\n",
    "    #SBATCH -e timing.err\n",
    "    #SBATCH -N 1\n",
    "    #SBATCH -c 4\n",
    "    #SBATCH -t 0-00:20\n",
    "    #SBATCH -p general\n",
    "    #SBATCH --mem-per-cpu 8000\n",
    "\n",
    "    source new-modules.sh\n",
    "    module load matlab\n",
    "    srun -n 1 -c 4 matlab-default -nosplash -nodesktop -r parallel_timing_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit this job. It will take many minutes to finish. Do you get expected speed-up?\n",
    "\n",
    "In timing.out, you should see something like\n",
    "\n",
    "    ans =\n",
    "\n",
    "     Pool with properties:\n",
    "\n",
    "                Connected: true\n",
    "               NumWorkers: 4\n",
    "                  Cluster: local\n",
    "            AttachedFiles: {}\n",
    "              IdleTimeout: 30 minutes (30 minutes remaining)\n",
    "              SpmdEnabled: true\n",
    "\n",
    "    serial time: 7.635188; result: 1000000000\n",
    "    parallel time: 5.901599; result: 1000000000\n",
    "    parallel time: 3.516169; result: 1000000000\n",
    "    Parallel pool using the 'local' profile is shutting down.\n",
    "\n",
    "Explain why the second `parfor` is faster then the first `parfor`\n",
    "\n",
    "**Tips**: Using batch job for this kind of small computation is definitely an overkill, as queuing and initializing would take much longer than actual compuation. You will probably use the interactive mode much more often in this class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus task: make your terminal prettier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open ~/.bash_profile (for example `vim ~/.bash_profile`), add the following lines\n",
    "\n",
    "For Mac\n",
    "\n",
    "    export CLICOLOR=1\n",
    "    export LSCOLORS=ExFxBxDxCxegedabagacad\n",
    "\n",
    "For Linux (Odyssey)\n",
    "\n",
    "    alias ls=\"ls --color=auto\"\n",
    "    \n",
    "Type `source ~/.bash_profile` or relaunch the terminal. Notice any difference? "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": false,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {
    "height": "239px",
    "left": "1px",
    "right": "20px",
    "top": "105px",
    "width": "212px"
   },
   "toc_section_display": "none",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}