Tutorials/Bash/Bash_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Very very Basic BASH tutorial\n",
    "\n",
    "This very basic tutorial is to introcuce a complete novice in the use of bash. The idea is that the person will be able to move around the Compute canada Clusters and be more or less comfortable with it. It is by no means comprehensive nor is going to make you a bash ninja, but it will allow you to deal with clusters, and some minor manipulations.\n",
    "\n",
    "\n",
    "## Don't be afraid of the terminal\n",
    "\n",
    "The terminal is your friend, and you'll see why. The terminal is the way we talk directly to the computer without windows and cliking. It looks something like this:\n",
    "\n",
    "<img src=\"files/images/Terminal.png\" width=\"500px\">\n",
    "\n",
    "If you work in windows you will need to install an ssh-enabled terminal like [mobaxterm](https://mobaxterm.mobatek.net/) or [putty](https://www.putty.org/). This tutorial will asume you already have installed and read the instructions of your terminal.\n",
    "\n",
    "For MAC and linux users, just search for terminal, and voila!\n",
    "\n",
    "##### TRY TO AVOID FILENAMES WITH SPACES IN THE TERMINAL...IS ANNOYING\n",
    "\n",
    "## SSH: Connecting to the cluster\n",
    "Since most of this tutorial is meant to help you use the Compute Canada clusters, I'll start with the connection. SSH stands for Secure SHell (shell is the terminal), and you just have to type `ssh username@clustername.computecanada.ca`, where `username` is your own username in Compute Canada and `clustername` is the name of the cluster. I believe most of use use Graham or Cedar. In this tutorial I'll assume we will be working in graham. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Pseudo-terminal will not be allocated because stdin is not a terminal.\r\n",
      "ssh: Could not resolve hostname graham.computecanada: Name or service not known\r\n"
     ]
    }
   ],
   "source": [
    "ssh jshleap@graham.computecanada.ca"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have typed your password, you are in the cluster!!!\n",
    "\n",
    "\n",
    "## Moving around the cluster\n",
    "If you only have a single Role (seehttps://ccdb.computecanada.ca/me/faq#what_is_role for the definition) and if you have never worked in this account before you should have two folders inside your account. To see the folders or the content of anything, we will need to ask the terminal to list the contents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bash Tutorial.ipynb  \u001b[0m\u001b[01;34mimages\u001b[0m/  README.md  Slurm_vs_PBS.ipynb\r\n"
     ]
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To move to a directory you need to use the change directory command `cd`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jshleap/my_gits/CristescuLab_misc/images\n"
     ]
    }
   ],
   "source": [
    "cd images\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0m\u001b[01;35mTerminal.png\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "ls\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute Canada clusters work with Unix operating systems (Not windows). This means that there are a few different things. The way the terminal knows where you want to go i thorugh the path. Paths are just a road map to the location you want to be in.  For example, lets imagine that we would like to go to a subdirectory called tutorial from my hme directory, there are two ways to do this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Errno 2] No such file or directory: '/home/tutorial'\n",
      "/home/jshleap/my_gits/CristescuLab_misc/images\n"
     ]
    }
   ],
   "source": [
    "cd /home/jshleap/tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Errno 2] No such file or directory: '/home/jshleap/tutorial'\n",
      "/home/jshleap/my_gits/CristescuLab_misc/images\n"
     ]
    }
   ],
   "source": [
    "cd ~/tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tilde (~) is a code for your home directory. In this notebook it fails because in my desktop this folder does not exist. So let's create one!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir ~/tutorial # this command tell the computer to create a folder named tutorial in your home directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now we can move to that folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/jshleap/tutorial\n"
     ]
    }
   ],
   "source": [
    "cd ~/tutorial"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you list the contents here with `ls` as before, you will notice that it is an empty folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets create an empty file using `touch`. This is not the most useful command, but allow you to know if you can write in the folder you are in, among other minial things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "touch afile.txt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if you type `ls` again, the afile.txt should appear. Lets try to edit this file. There are many tools, and depending on how you are sshing into the clusters they might change. I will focus in the native reader of unix **nano**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nano afile.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see something like this:\n",
    "<img src=\"files/images/nano.png\" width=\"500px\">\n",
    "You can write anything you want there. Lets try just typing \"this is a test file\". To close and save you have to press the ``ctrl + x``. It will ask you if you want to save what you just write, type `Y` and `enter`.\n",
    "So you might be asking if you have to open nano everytime you need to read the file without modyfying it? The answer is no. There are a few commands you can use:\n",
    "* cat if your file is small\n",
    "* head if you want to see the first few lines of your file\n",
    "* tail if you want to see the last few lines of your file\n",
    "If you execute `cat afile.txt` it will show you what you just wrote. Cat will output the entire content of the file to screen. That is why is only wise to use it in small files or for certain manipulations (outside of the scope of this tutorial). The same way we can use head `head afile.txt` and tail `tail afile.txt`. In the litle single line file, all three commands should show exactly the same.\n",
    "\n",
    "Now let's remove that file with the command `rm`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm afile.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Be very careful with the usage of `rm`, since **once you have use it in a file you cannot recover the file**.\n",
    "\n",
    "## Downloading from the web\n",
    "Now, we need a more substantial file to play around with, so let's download one from the web directly into the cluster!!! For this we will use the command `wget`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wget https://raw.githubusercontent.com/jshleap/CristescuLab_misc/master/Tutorials/Bash/test_files/allponds.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will download a fasta file with some sequences. As you probably know, a fasta file (!!!NOT A FASTQ) is formated like:\n",
    "```\n",
    ">SEQNAME OR INFO FOR SEQ1\n",
    "SEQUENCE\n",
    ">SEQNAME OR INFO FOR SEQ1\n",
    "SEQUENCE\n",
    "```\n",
    "\n",
    "## Exploring a file\n",
    "Now let's explore this file with the `cat`, `head`, and `tail` commands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat allponds.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head allponds.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head -n 5 allponds.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `-n` flag in head tells the command how many lines to output!! Play around with it! The `tail` command works very similar but instead of showing the first `n` lines, it shows the `n` last:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tail -n 5 allponds.fasta # play around with it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Explore the file page by page\n",
    "Let's say that you'd like to see the file page by page instead of the whole thing at once and with more than a given number of lines. You can do this with the comands `less` and `more`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "more allponds.fasta # to quit press Q,  to move return or space bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "less allponds.fasta # to quit press Q to move use arrows or space bar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's imagine that you would like to see the total number of lines that this file has, you can use the word count command `wc`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 1000  1200 67931 ./test_files/allponds.fasta\r\n"
     ]
    }
   ],
   "source": [
    "wc allponds.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tells us that the file has 1000 lines, 1200 words and 67931 characters. If we want just the number of lines we can use `wc -l`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000 ./test_files/allponds.fasta\r\n"
     ]
    }
   ],
   "source": [
    "wc -l allponds.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great!! now we know that we have a 1000 lines file, but how many sequences do we have? Because allponds.fasta is a fasta (**not a fastq**), the sequences are indicated by the `>` symbol. This should be a unique symbol in fasta BUT NOT IN FASTQ. We will need to search occurencies of this symbol. Fortunately, bash have the very handy command `grep'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">M00833:667:000000000-BV9MW:1:2106:28755:11977 1:N:0:TAAGGCGA+GCGTAAGA;size=685;\r\n",
      ">M00833:667:000000000-BV9MW:1:2107:4035:17001 1:N:0:TAAGGCGA+GCGTAAGA;size=510;\r\n",
      ">M00833:667:000000000-BV9MW:1:2106:27458:12091 1:N:0:TAAGGCGA+GCGTAAGA;size=431;\r\n",
      ">M00833:667:000000000-BV9MW:1:2106:17802:12685 1:N:0:TAAGGCGA+GCGTAAGA;size=268;\r\n",
      ">M00833:667:000000000-BV9MW:1:2106:23093:11024 1:N:0:TAAGGCGA+GCGTACGA;size=217;\r\n",
      ">M00833:667:000000000-BV9MW:1:2107:17986:18261 1:N:0:TAAGGCGA+GCGTAAGA;size=200;\r\n",
      ">M00833:667:000000000-BV9MW:1:2107:24169:15780 1:N:0:TAAGGCGA+GCGTAAGA;size=198;\r\n",
      ">M00833:667:000000000-BV9MW:1:2106:7985:12895 1:N:0:TAAGGCGA+GCGTAAGA;size=163;\r\n",
      ">M00833:667:000000000-BV9MW:1:2109:9233:9403 1:N:0:TAAGGCGA+GCGTAAGA;size=137;\r\n",
      ">M00833:667:000000000-BV9MW:1:2107:19829:15122 1:N:0:TAAGGCGA+GCGTAAGA;size=99;\r\n"
     ]
    }
   ],
   "source": [
    "grep '>' allponds.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, this gave us all the lines where `>` is found in the file, which is useful, but not what we where looking for. We would like to count the lines to know the number of sequences we have. Fortunately, grep has the -c option:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\r\n"
     ]
    }
   ],
   "source": [
    "grep -c '>' allponds.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perfect!! it tell us that there are 200!!! Can this fail? YES! if there are more `>` in the file in lines that are not the sequence. However, if you had a `>` is the sequence of a fasta file, it would be a wrongly formatted file. Why am I telling you this? beacuse dealing with fastq files is a different story. I will get back at this later in the tutorial, but keep it in mind.\n",
    "\n",
    "\n",
    "### To recap:\n",
    "1. To list all files and directories use `ls`\n",
    "2. To move to a given directory use `cd`\n",
    "3. To create a folder use `mkdir`\n",
    "4. To remove a **file** use `rm`\n",
    "5. To create an empty file use `touch`\n",
    "6. To natively edit a file use `nano`\n",
    "7. To explore a file page by page use `less` or `more`\n",
    "8. To count the number of lines, words and characters in a file use `wc`\n",
    "9. To count exclusively the number of lines in a file use `wc -l`\n",
    "10. To find a pattern in a file use `grep`\n",
    "11. To find a pattern in a file and count the occurences use `grep -c`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regular expressions, redirection, and pipes\n",
    "OK, so far we can move from one directory to another, we can create directories and files, remove files, and find characters and words in a file. But there are a bunch of useful things that we can use these same commands that expands their usability. One of these things is the usage of regular expresions. According to [wikipedia](https://en.wikipedia.org/wiki/Regular_expression) is *\"a sequence of characters that define a search pattern\"*. With some special characters we can expand searches for example. One of the most used one is the wildcard `*`. This means that in that position it can be anything. Let's use it with the list `ls` command, but first lets create a bunch of files with a given pattern:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#this is a loop. We are not covering this in this tutorial, but have a look if you are interested\n",
    "for i in {1..10}; do touch prefix${i}.txt;done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code above will create 10 files. list them using `ls`. Now with only `ls` you can see all files, including `allponds.fasta` and the previous files that we have created. Now, if we would like to list only the files that end in `.txt`, we can:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls *.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This command tells the computer to replace `*` for anything (ergo wildcard), and therefore list anything that ends in `.txt`. But if we have multiple `.txt` files and you would like to list a particular pattern, you can also use the wildcard:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls prefix*.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This just list all the files that start with `prefix`, and end with `.txt`, no matter what it is in the middle. This strategy can be used with most commands. To test, write something in each of the 10 files, and then lets do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat prefix*.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It must print to screen a concatenation of all the files you wrote on. Now, what if you would like to merge all this files? well we can use the same strategy as before but we will **redirect** the output to a file, using the redirection command `>`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat prefix*.txt > merged.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the contents of the new file! \n",
    "Another useful regex (a.k.a. regular expresion) that is useful is the `^` symbol. It means the begining of a line! As an example, let's download a fastq file and try to count the lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wget https://raw.githubusercontent.com/jshleap/CristescuLab_misc/master/Tutorials/Bash/test_files/example.fastq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explore the file with the tools you now know how to use!! Place special attention to the symbol `@`. In a fastq file, `@` refers both to the sequence ID, as well as a quality Score when the `@` is at the quality section. Because of this we cannot count the number of sequences by a plain `grep -c '@' example.fastq`. If we do, we will be counting also the quality line of the file. Try grepping the `@` without the `-c` flag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep '@' example.fastq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Difficult to see everything? No problem! there is another bash trick, PIPES! You can feed one command with the output of other command. Let's try to get the fist 10 lines of the grep output we just had. We saw that head does that with a file, but if we want head to get the output of grep we use the pipe (`|`) command like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep '@' example.fastq | head -n 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It also works with `more` or `less` commands!:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep '@' example.fastq | more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking the file you would see that if we use `grep '@'` we will double count in many parts of the file. It would be better if we could grep only when `'@'` is at the begining of the line. To do that we go back to the regular expresion `^`. If we do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep '^@' example.fastq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that most of all of our output starts with `@` and most of it corresponds to the sequences. However some instances of our output will contain non-seqid lines, so counting will over estimate the number of sequences in the fastq file. To resolve this we need to find a pattern unique to the seqID. In this particular case, all sequences start with `M`. Let's explore this with `grep` and regex:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep '^@M' example.fastq | more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we only see sequences IDs. So how many sequences we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep -c '^@M' example.fastq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a more realistic count of our sequences. **NOTE: MAKE SURE YOUR PATTERN IS CONSISTENT TO AVOID MISCOUNTS**.\n",
    "\n",
    "### Sorting files and getting unique counts\n",
    "Now, lets imaging we have duplicate IDs. Let's create this scenario by concatenating twice the `example.fastq`, and redirecting the output to `double.fastq:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat example.fastq example.fastq > double.fastq\n",
    "grep -c '^@M' double.fastq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This must give you twice as the number before. Now let's imagine we would like to have a list of IDs with unique identifyers. Fortuantely, bash have the right tool for it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep '^@M' double.fastq | sort -u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first part of the command `grep '^@M' double.fastq` will get the seqids as before, and the output would be piped into `sort -u`, which will sort the output, and return only the unique entries. Now let's imagine we dont want the entire output, but just the count, so our good old friend `wc -l` will come handy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep '^@M' double.fastq | sort -u | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic introduction to jobs in the cluster\n",
    "Because a supercomputer or cluster is a collection of \"blades\" or individual computer connected to each other, there is a specific environment to make it work. For example, programs need to be installed in a way that would be accesible for every node. For this, Compute Canada uses the modules enviroment. This means that you will have to load the program (if available) that you would like to run. How can you know what is installed? well, there is a command for that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "module avail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This command will give you a list of all available programs in the cluster. Normally, there are quite a bit of files and it is difficult to identify them all at once. So there is a second useful `module` command: `module spider`. Let's imagine we would like to use the command line version of blast. This suite is called blast+, so we can type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "module spider blast+"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will show you all the entries and versions for blast+, along with some important information of pre-requisites you have to load before blast+. Lets take a look to them. In this case, it tells use that before we load blast+ we need the modules `nixpkgs/16.09`, and  `gcc/5.4.0`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "module load nixpkgs/16.09 gcc/5.4.0 blast+"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will load the three modules we need. Let's see if it worked by typing `blastn -h`. Now we have the blast commands we need to run a job. However, **and this is very important**, DO NOT RUN JOBS ON THE LOGING NODE. If you do, you most likely will received a very angry email and your job will be killed before it finishes. This is done beacuse the login node is where eveybody launches their jobs to the rest of the computers, so by running something there it slows everybody down. To send jobs to the compute nodes you need `sbatch` and a launching script. I will get to that a bit later. Now, let's imagine that we need to do a smaller quick test on the spot. For that we will invoke the interactive shell using the `salloc` command. This command will allocate us some space in a compute node. To call this, we require the account that we are registered in compute canada. For Cristescu lab members it would be `def-mcristes`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "salloc -A def-mcristes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where `-A` stands for account. Actually, you can also call it as `--account=def-mcristes`. Once it starts we can play around with the blast commands. This tutorial will cover only one basic nucleotide blast. Since we do not have downloaded the databses (extremely recomended if you do a lot of blasting), we will use the remote option. This option takes a bit more, for is ok for this tutorial:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "blastn -db nt -query allponds.fasta -outfmt '6 sseqid qseqid evalue pident qcovs stitle' -remote -max_target_seqs 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the `-db nt` tells the program to use the nucleotide database, the `-query allponds.fasta` is the query sequences we want to search against the NCBI datase, `-outfmt` is the format of the output and we put 6(tabular), and restricting the output to sseqid, qseqid, evalue, pident, qcovs, and stitle. The option `-remote` esentially tells the program to do it over the internet instead of with a particular database. The `-max_target_seqs 2` makes sure we retained only the top 2 entries of the search.\n",
    "\n",
    "If you are interested in this tool, check out their manual at [NCBI](https://www.ncbi.nlm.nih.gov/books/NBK279690/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to finish our session of `salloc`, type `exit`. This will bring us back to the login node. Now to actually use more demanding (both in memory and in time) tasks, we will need to submit a script to the cluster. Most of the basics are already covered in the Compute Canada wikis like [this one](https://docs.computecanada.ca/wiki/Running_jobs). First we need to create a script to launch the job, and it has a particular structure and keywords. Here is an example:\n",
    "```\n",
    "#!/bin/bash\n",
    "#SBATCH --time=00:01:00\n",
    "#SBATCH --account=def-mcristes\n",
    "echo 'Hello, world!'\n",
    "sleep 30\n",
    "```\n",
    "If you launch this script, it will send it to a computing node for no more than 1 minute. So let's create our own!! Copy this template, but replace the last two lines with something you would like. Also, try to redirect the output to a file named `myecho.txt`.\n",
    "\n",
    "If what you are going to run is, for example, the previous blast run, the file should look like this:\n",
    "\n",
    "```\n",
    "#!/bin/bash\n",
    "#SBATCH --time=00:10:00\n",
    "#SBATCH --account=def-mcristes\n",
    "module load nixpkgs/16.09 gcc/5.4.0 blast+\n",
    "blastn -db nt -query allponds.fasta -outfmt '6 sseqid qseqid evalue pident qcovs stitle' -remote -max_target_seqs 2\n",
    "```\n",
    "See how the module call is included?\n",
    "\n",
    "\n",
    "## WORK IT OUT!!\n",
    "I have shown you the very basics! Now is time for you to do it on your own! \n",
    "\n",
    "1. Go to your home directory \n",
    "2. Create a folder (name it as you wish)\n",
    "3. Go into that folder\n",
    "4. Go to the NCBI website, select your favorite gene, in the make sure you selected FASTA (text). \n",
    "5. Use wget to download the sequence into your folder\n",
    "6. Create a submission script and use blast remote as in the example, but with this query\n",
    "7. Get the results in a file named myfirstclusterblast.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Well, that is all for now, let's work a little bit on this and I'll take questions you might still have!!! cheers!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}