01-bash/01-introducing-the-shell_instructor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introducing the Shell "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Background\n",
    "At a high level, computers do four things:\n",
    "\n",
    "-   run programs\n",
    "-   store data\n",
    "-   communicate with each other, and\n",
    "-   interact with us\n",
    "\n",
    "They can do the last of these in many different ways,\n",
    "including through a keyboard and mouse, or touch screen interfaces, or speech recognition using systems.\n",
    "While such hardware interfaces are becoming more commonplace, most interaction is still\n",
    "done using screens, mice, touchpads and keyboards.\n",
    "\n",
    "We are all familiar with **graphical user interfaces** (GUI): windows, icons and pointers.\n",
    "They are easy to learn and fantastic for simple tasks where a vocabulary consisting of\n",
    "\"click\" translates easily into \"do the thing I want\". But this magic relies on \n",
    "wanting a simple set of things, and having programs that can do exactly those things.\n",
    "\n",
    "If you wish to do complex, purpose-specific things it helps to have a richer means\n",
    "of expressing your instructions to the computer. It doesn't need to be complicated or\n",
    "difficult, just a vocabulary of commands and a simple grammar for using them.\n",
    "\n",
    "This is what the shell provides - a simple language and a **command-line interface** \n",
    "to use it through. \n",
    "\n",
    "The heart of a command-line interface is a **read-evaluate-print loop** (REPL) called\n",
    "so because when you type a command and press <kbd>Return</kbd>.\n",
    "The shell\n",
    "reads it,\n",
    "executes (or \"evaluates\" it),\n",
    "prints the output,\n",
    "prints the prompt and waits for you to enter another command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Shell\n",
    "\n",
    "A shell is a program like any other.\n",
    "What's special about it is that its job is to run other programs\n",
    "rather than to do calculations itself.\n",
    "The most popular Unix shell is Bash,\n",
    "the Bourne Again SHell\n",
    "(so-called because it's derived from a shell written by Stephen Bourne).\n",
    "Bash is the default shell on most modern implementations of Unix\n",
    "and in most packages that provide Unix-like tools for Windows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What does it look like?\n",
    "\n",
    "A typical shell command looks something like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -F / "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first line shows only a **prompt**,\n",
    "indicating that the shell is waiting for input.\n",
    "Your shell may use different text for the prompt. Most importantly: \n",
    "when typing commands, either from these lessons or from other sources,\n",
    "*do not type the prompt*, only the commands that follow it.\n",
    "\n",
    "The part that you type,\n",
    "`ls -F /` in the second line of the example,\n",
    "typically has the following structure: a **command**,\n",
    "some **flags** (also called **options** or **switches**) and an **argument**.\n",
    "Flags start with a single dash (`-`) or two dashes (`--`), and change the behaviour of a command.\n",
    "Arguments tell the command what to operate on (e.g. files and directories).\n",
    "Sometimes flags and arguments are referred to as parameters.\n",
    "A command can be called with more than one flag and more than one argument: but a\n",
    "command doesn't always require an argument or a flag.\n",
    "\n",
    "In the second line of the example above, our **command** is `ls`, with a **flag** `-F` and an\n",
    "**argument** `/`. Each part is separated by spaces: if you omit the space \n",
    "between `ls` and `-F` the shell will look for a command called `ls-F`, which \n",
    "doesn't exist. Also, capitalization matters: `LS` is different to `ls`. \n",
    "\n",
    "Next we see the output that our command produced. In this case it is a listing \n",
    "of files and folders in a location called `/` - we'll cover what all these mean \n",
    "later today. Those using a macOS might recognize the output in this example.\n",
    "\n",
    "Finally, the shell again prints the prompt and waits for you to type the next \n",
    "command.\n",
    "\n",
    "In the examples for this lesson, we'll show the prompt as `$ `. You can make your \n",
    "prompt look the same by entering the command `PS1='$ '`. But you can also leave \n",
    "your prompt as it is - often the prompt includes useful information about who and where \n",
    "you are.\n",
    "\n",
    "Open a shell window and try entering `ls -F /` for yourself (don't forget that spaces\n",
    "and capitalization are important!). You can change the prompt too, if you like."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How does the shell know what `ls` and its flags mean?\n",
    "\n",
    "Every command is a program stored somewhere on the computer, and the shell keeps a\n",
    "list of places to search for commands (the list is in a **variable** called `PATH`, \n",
    "but those are concepts we'll meet later and not too important at the moment). Recall\n",
    "that commands, flags and arguments are separated by spaces.\n",
    "\n",
    "So let's look at the REPL (read-evaluate-print loop) in more detail. Notice that the\n",
    "\"evaluate\" step is made of two parts: \n",
    "\n",
    "1. Read what was typed (`ls -F /` in our example)  \n",
    "    The shell uses the spaces to split the line into the command, flags, and arguments\n",
    "2. Evaluate:  \n",
    "    a. Find a program called `ls`  \n",
    "    b. Execute it, passing it the flags and arguments (`-F` and `/`) to \n",
    "       interpret as the program sees fit \n",
    "3. Print the output produced by the program\n",
    "\n",
    "and then print the prompt and wait for you to enter another command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<section class=\"callout panel panel-warning\">\n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-thumb-tack\"></span> Command not found</h2>\n",
    "</div>\n",
    "\n",
    "\n",
    "<div class=\"panel-body\">\n",
    "\n",
    "<p>If the shell can't find a program whose name is the command you typed, it \n",
    "will print an erorr message like:</p>\n",
    "<div class=\"codehilite\"><pre><span></span>ls-F\n",
    "</pre></div>\n",
    "\n",
    "\n",
    "<div class=\"codehilite\"><pre><span></span>-bash: ls-F: command not found\n",
    "</pre></div>\n",
    "\n",
    "\n",
    "<p>Usually this means that you have mis-typed the command - in this case we omitted\n",
    "the space between <code>ls</code> and <code>-F</code>. </p>\n",
    "\n",
    "</div>\n",
    "\n",
    "</section>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Is it difficult?\n",
    "\n",
    "It is a different model of interacting than a GUI, and that \n",
    "will take some effort - and some time - to learn. A GUI \n",
    "presents you with choices and you select one. With a **command line interface** (CLI) the choices are combinations \n",
    "of commands and parameters, more like words in a language than buttons on a screen. They\n",
    "are not presented to you so\n",
    "you must learn a few, like learning some vocabulary in a new language. But a small \n",
    "number of commands gets you a long way, and we'll cover those essential few today."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flexibility and automation \n",
    "\n",
    "The grammar of a shell allows you to combine existing tools into powerful\n",
    "pipelines and handle large volumes of data automatically. Sequences of\n",
    "commands can be written into a *script*, improving the reproducibility of \n",
    "workflows and allowing you to repeat them easily.\n",
    "\n",
    "In addition, the command line is often the easiest way to interact with remote machines and supercomputers.\n",
    "Familiarity with the shell is near essential to run a variety of specialized tools and resources\n",
    "including high-performance computing systems.\n",
    "As clusters and cloud computing systems become more popular for scientific data crunching,\n",
    "being able to interact with the shell is becoming a necessary skill.\n",
    "We can build on the command-line skills covered here\n",
    "to tackle a wide range of scientific questions and computational challenges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nelle's Pipeline: Starting Point\n",
    "\n",
    "Nelle Nemo, a marine biologist,\n",
    "has just returned from a six-month survey of the\n",
    "[North Pacific Gyre](http://en.wikipedia.org/wiki/North_Pacific_Gyre),\n",
    "where she has been sampling gelatinous marine life in the\n",
    "[Great Pacific Garbage Patch](http://en.wikipedia.org/wiki/Great_Pacific_Garbage_Patch).\n",
    "She has 1520 samples in all and now needs to:\n",
    "\n",
    "1.  Run each sample through an assay machine\n",
    "    that will measure the relative abundance of 300 different proteins.\n",
    "    The machine's output for a single sample is\n",
    "    a file with one line for each protein.\n",
    "2.  Calculate statistics for each of the proteins separately\n",
    "    using a program her supervisor wrote called `goostats`.\n",
    "3.  Write up results.\n",
    "    Her supervisor would really like her to do this by the end of the month\n",
    "    so that her paper can appear in an upcoming special issue of *Aquatic Goo Letters*.\n",
    "\n",
    "It takes about half an hour for the assay machine to process each sample.\n",
    "The good news is that\n",
    "it only takes two minutes to set each one up.\n",
    "Since her lab has eight assay machines that she can use in parallel,\n",
    "this step will \"only\" take about two weeks.\n",
    "\n",
    "The bad news is that if she has to run `goostats` by hand,\n",
    "she'll have to enter filenames and click \"OK\" 1520 times.\n",
    "At 30 seconds per sample,\n",
    "the whole process will take more than 12 hours\n",
    "(and that's assuming the best-case scenario where she is ready to enter the next file name\n",
    "as soon as the previous sample analysis has finished).\n",
    "This zero-breaks always-ready scenario is only achieveable by a machine so it would\n",
    "likely take much longer than 12 hours, not to mention that\n",
    "the chances of her typing all of those commands correctly are practically zero.\n",
    "Missing that paper deadline is looking increasingly likely.\n",
    "\n",
    "The next few lessons will explore what she should do instead.\n",
    "More specifically,\n",
    "they explain how she can use a command shell\n",
    "to automate the repetitive steps in her processing pipeline\n",
    "so that her computer can work 24 hours a day while she writes her paper.\n",
    "As a bonus,\n",
    "once she has put a processing pipeline together,\n",
    "she will be able to use it again whenever she collects more data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<section class=\"keypoints panel panel-success\">\n",
    "<div class=\"panel-heading\">\n",
    "<h2><span class=\"fa fa-exclamation-circle\"></span> Keypoints</h2>\n",
    "</div>\n",
    "\n",
    "\n",
    "<div class=\"panel-body\">\n",
    "\n",
    "<ul>\n",
    "<li>A shell is a program whose primary purpose is to read commands and run other programs.</li>\n",
    "<li>The shell's main advantages are its high action-to-keystroke ratio, its support for automating repetitive tasks, and its capacity to access networked machines.</li>\n",
    "<li>The shell's main disadvantages are its primarily textual nature and how cryptic its commands and operation can be.</li>\n",
    "</ul>\n",
    "\n",
    "</div>\n",
    "\n",
    "</section>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "The material in this notebook is derived from the Software Carpentry lessons\n",
    "&copy; [Software Carpentry](http://software-carpentry.org/) under the terms\n",
    "of the [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/) license."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  },
  "swc_attribution": true
 },
 "nbformat": 4,
 "nbformat_minor": 2
}