Intro to Shell for Data Science - solutions.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Intro to Shell for Data Science - solutions.ipynb","version":"0.3.2","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"code","metadata":{"id":"NF-nL4WUsW3G","colab_type":"code","colab":{}},"source":["# https://www.datacamp.com/courses/introduction-to-shell-for-data-science"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5XyAVVOSscMZ","colab_type":"text"},"source":["## 1. Manipulating files and directories"]},{"cell_type":"markdown","metadata":{"id":"5ImMuaKDtLbQ","colab_type":"text"},"source":["**Where am I?**\n","\n","To find out where you are in the filesystem, run the command pwd (short for \"print working directory\"). This prints the absolute path of your current working directory, which is where the shell runs commands and looks for files by default.\n"]},{"cell_type":"code","metadata":{"id":"GxAazSiQsefB","colab_type":"code","colab":{}},"source":["! pwd"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zpEKfpuatYRp","colab_type":"text"},"source":["**How can I identify files and directories?**\n","\n","To find out what's there, type ls (which is short for \"listing\") and press the enter key. On its own, ls lists the contents of your current directory (the one displayed by pwd). If you add the names of some files, ls will list them, and if you add the names of directories, it will list their contents."]},{"cell_type":"code","metadata":{"id":"R6daQOeytXtb","colab_type":"code","colab":{}},"source":["! ls"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"6twpacKMt1XE","colab_type":"text"},"source":["**How else can I identify files and directories?**\n","\n","The shell decides if a path is absolute or relative by looking at its first character: if it begins with /, it is absolute, and if it doesn't, it is relative."]},{"cell_type":"markdown","metadata":{"id":"SaGvrnr3LtGK","colab_type":"text"},"source":["1. You are in /home/repl. Use ls with a relative path to list the file /home/repl/course.txt (and only that file)."]},{"cell_type":"code","metadata":{"id":"RNEfHZr1t-UD","colab_type":"code","colab":{}},"source":["! ls course.txt "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"12_lx1CYL4-5","colab_type":"text"},"source":["2. You are in /home/repl. Use ls with a relative path to list the file /home/repl/seasonal/summer.csv (and only that file)."]},{"cell_type":"code","metadata":{"id":"KTcvF-RhL78B","colab_type":"code","colab":{}},"source":["! ls seasonal/summer.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"mLU2i99wMF-5","colab_type":"text"},"source":["3. You are in /home/repl. Use ls with a relative path to list the contents of the directory /home/repl/people."]},{"cell_type":"code","metadata":{"id":"pMcd2IZ2MG_I","colab_type":"code","colab":{}},"source":["! ls people"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FD0_L9n0uVu1","colab_type":"text"},"source":["**How can I move to another directory?**\n","\n","Just as you can move around in a file browser by double-clicking on folders, you can move around in the filesystem using the command cd (which stands for \"change directory\")."]},{"cell_type":"markdown","metadata":{"id":"m7BnKM1lMRXp","colab_type":"text"},"source":["1. You are in /home/repl/. Change directory to /home/repl/seasonal using a relative path."]},{"cell_type":"code","metadata":{"id":"QPUm0Uo0uaR1","colab_type":"code","colab":{}},"source":["! cd seasonal "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"9O5jJcn5MfBA","colab_type":"text"},"source":["2. Use pwd to check that you're there."]},{"cell_type":"code","metadata":{"id":"qfvclu23ukcy","colab_type":"code","colab":{}},"source":["! pwd"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"b1hWYfAAMimZ","colab_type":"text"},"source":["3. Use ls without any paths to see what's in that directory."]},{"cell_type":"code","metadata":{"id":"zA8vRIS9Mk25","colab_type":"code","colab":{}},"source":["! ls"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Fb0UF-Gfu3G2","colab_type":"text"},"source":["**How can I move up a directory?**\n","\n","The parent of a directory is the directory above it. \n","\n",". You can always give the absolute path of your parent directory to commands like cd and ls. More often, though, you will take advantage of the fact that the special path .. (two dots with no spaces) means \"the directory above the one I'm currently in\". \n","\n","A single dot on its own, ., always means \"the current directory\", so ls on its own and ls . do the same thing, while cd . has no effect (because it moves you into the directory you're currently in).\n","\n","One final special path is ~ (the tilde character), which means \"your home directory\", such as /home/repl. No matter where you are, ls ~ will always list the contents of your home directory, and cd ~ will always take you home."]},{"cell_type":"code","metadata":{"id":"GcfYKbtbvAOD","colab_type":"code","colab":{}},"source":["! cd ~/../. # The path means 'home directory', 'up a level', 'here'."],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"jgIorD8UvRgg","colab_type":"text"},"source":["**How can I copy files?**\n","\n","You will often want to copy files, move them into other directories to organize them, or rename them. One command to do this is cp, which is short for \"copy\"."]},{"cell_type":"markdown","metadata":{"id":"qLAKxMNSMza4","colab_type":"text"},"source":["1. Make a copy of seasonal/summer.csv in the backup directory (which is also in /home/repl), calling the new file summer.bck."]},{"cell_type":"code","metadata":{"id":"Wckry5qKvOJ4","colab_type":"code","colab":{}},"source":["! cp seasonal/summer.csv backup/summer.bck"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"BoP40hiINQ9Y","colab_type":"text"},"source":["2. Copy spring.csv and summer.csv from the seasonal directory into the backup directory without changing your current working directory (/home/repl)."]},{"cell_type":"code","metadata":{"id":"ofs-fyuKNTOH","colab_type":"code","colab":{}},"source":["! cp seasonal/spring.csv seasonal/summer.csv backup"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qN9X1NI8NdH_","colab_type":"text"},"source":["**How can I move a file?**\n","\n","While cp copies a file, mv moves it from one directory to another, just as if you had dragged it in a graphical file browser. \n","\n","- You are in /home/repl, which has sub-directories seasonal and backup. Using a single command, move spring.csv and summer.csv from seasonal to backup."]},{"cell_type":"code","metadata":{"id":"ShcSY9RrOYju","colab_type":"code","colab":{}},"source":["! mv seasonal/spring.csv seasonal/summer.csv backup"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"olS751DwOnEW","colab_type":"text"},"source":["**How can I rename files?**\n","\n","mv can also be used to rename files. If you run:\n","\n","mv course.txt old-course.txt\n","then the file course.txt in the current working directory is \"moved\" to the file old-course.txt. \n","\n","*One warning: just like cp, mv will overwrite existing files. If, for example, you already have a file called old-course.txt, then the command shown above will replace it with whatever is in course.txt.*"]},{"cell_type":"markdown","metadata":{"id":"YlTEDGRBOxqG","colab_type":"text"},"source":["1. Go into the seasonal directory."]},{"cell_type":"code","metadata":{"id":"byTXIgoGOuam","colab_type":"code","colab":{}},"source":["! cd seasonal"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"3mTVl56GO1JN","colab_type":"text"},"source":["2. Rename the file winter.csv to be winter.csv.bck."]},{"cell_type":"code","metadata":{"id":"x_eHcbc9PBBF","colab_type":"code","colab":{}},"source":["! mv winter.csv winter.csv.bck"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qCIWTGDfPHht","colab_type":"text"},"source":["3. Run ls to check that everything has worked."]},{"cell_type":"code","metadata":{"id":"5eG3hO2BPJsl","colab_type":"code","colab":{}},"source":["! ls"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"2I2wwf7mPON9","colab_type":"text"},"source":["**How can I delete files?**\n","\n","We can copy files and move them around; to delete them, we use rm, which stands for \"remove\". As with cp and mv, you can give rm the names of as many files as you'd like."]},{"cell_type":"markdown","metadata":{"id":"U988uzSGPXE-","colab_type":"text"},"source":["1. You are in /home/repl. Go into the seasonal directory.\n","2. Remove autumn.csv.\n","3. Go back to your home directory.\n","4. Remove seasonal/summer.csv without changing directories again."]},{"cell_type":"code","metadata":{"id":"JKKV584GPaj1","colab_type":"code","colab":{}},"source":["! cd seasonal"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"mVeXMo6OPgOO","colab_type":"code","colab":{}},"source":["! rm autumn.csv"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"WtPHioMXPmte","colab_type":"code","colab":{}},"source":["! cd .. # or cd ~"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"XWzMeRIDPtwF","colab_type":"code","colab":{}},"source":["! rm seasonal/summer.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"I4WcFAiNQHf8","colab_type":"text"},"source":["**How can I create and delete directories?**\n","\n","mv treats directories the same way it treats files: if you are in your home directory and run mv seasonal by-season, for example, mv changes the name of the seasonal directory to by-season. However, rm works differently.\n","\n","If you try to rm a directory, the shell prints an error message telling you it can't do that, primarily to stop you from accidentally deleting an entire directory full of work. Instead, you can use a separate command called rmdir. For added safety, it only works when the directory is empty, so you must delete the files in a directory before you delete the directory."]},{"cell_type":"markdown","metadata":{"id":"h5l32z4RRlVT","colab_type":"text"},"source":["1. Without changing directories, delete the file agarwal.txt in the people directory."]},{"cell_type":"code","metadata":{"id":"MjEyffXpRxFS","colab_type":"code","colab":{}},"source":["! rm people/agarwal.txt "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zr9B8vmfSFH6","colab_type":"text"},"source":["2. Now that the people directory is empty, use a single command to delete it."]},{"cell_type":"code","metadata":{"id":"wwIFBN9ASF6T","colab_type":"code","colab":{}},"source":["! rmdir people"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"4xKuONlcSLa6","colab_type":"text"},"source":["3. Since a directory is not a file, you must use the command mkdir directory_name to create a new (empty) directory. Use this command to create a new directory called yearly below your home directory."]},{"cell_type":"code","metadata":{"id":"3oPcAs8WSMSb","colab_type":"code","colab":{}},"source":["! mkdir yearly"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"8Dwc3AhfSUlr","colab_type":"text"},"source":["4. Now that yearly exists, create another directory called 2017 inside it without leaving your home directory."]},{"cell_type":"code","metadata":{"id":"sjWqipx6SVYU","colab_type":"code","colab":{}},"source":["! mkdir yearly/2017 "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"rGCHG83LSfRS","colab_type":"text"},"source":["**Wrapping up**\n","\n","You will often create intermediate files when analyzing data. Rather than storing them in your home directory, you can put them in /tmp, which is where people and programs often keep files they only need briefly. (Note that /tmp is immediately below the root directory /, not below your home directory.) \n","\n","1. Use cd to go into /tmp\n","2. List the contents of /tmp without typing a directory name\n","3. Make a new directory inside /tmp called scratch.\n","4. Move /home/repl/people/agarwal.txt into /tmp/scratch. We suggest you use the ~ shortcut for your home directory and a relative path for the second rather than the absolute path."]},{"cell_type":"code","metadata":{"id":"DF8ZRSkuSm-y","colab_type":"code","colab":{}},"source":["! cd /tmp"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"s3FccN8bS8li","colab_type":"code","colab":{}},"source":["! ls"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"lactp-I5S9Pr","colab_type":"code","colab":{}},"source":["! mkdir scratch"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"dh6VtkFyTHHB","colab_type":"code","colab":{}},"source":["! mv ~/people/agarwal.txt scratch"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"nnbRRSW8seoY","colab_type":"text"},"source":["## 2. Manipulating data"]},{"cell_type":"markdown","metadata":{"id":"9n1Ywe_9viI0","colab_type":"text"},"source":["**How can I view a file's contents?**\n","\n","Before you rename or delete files, you may want to have a look at their contents. The simplest way to do this is with cat, which just prints the contents of files onto the screen. (Its name is short for \"concatenate\", meaning \"to link things together\", since it will print all the files whose names you give it, one after the other.)\n","\n","- Print the contents of course.txt to the screen."]},{"cell_type":"code","metadata":{"id":"Wl9zOlY8s_YZ","colab_type":"code","colab":{}},"source":["! cat course.txt"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Tq7Bvp5BwEae","colab_type":"text"},"source":["**How can I view a file's contents piece by piece?**\n","\n","You can use cat to print large files and then scroll through the output, but it is usually more convenient to page the output. The original command for doing this was called more, but it has been superseded by a more powerful command called less.\n","\n","\n","\n","*   Use less seasonal/spring.csv seasonal/summer.csv to view those two files in that order. Press spacebar to page down, :n to go to the second file, and :q to quit.\n","\n","\n","\n"]},{"cell_type":"code","metadata":{"id":"zOPKANfKv-4G","colab_type":"code","colab":{}},"source":["! less seasonal/spring.csv seasonal/summer.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"uDXdbi28wcNF","colab_type":"text"},"source":["**How can I look at the start of a file?**\n","\n"," A quick way to figure out what it contains is to look at the first few rows.\n","\n","We can do this in the shell using a command called head. As its name suggests, it prints the first few lines of a file (where \"a few\" means 10)."]},{"cell_type":"code","metadata":{"id":"x-m6sBVowvI8","colab_type":"code","colab":{}},"source":["! head seasonal/autumn.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"IvBhHxzkw4uA","colab_type":"text"},"source":["**How can I type less?**\n","\n","One of the shell's power tools is tab completion. If you start typing the name of a file and then **press the tab key**, the shell will do its best to auto-complete the path. "]},{"cell_type":"markdown","metadata":{"id":"qm0w6FG9xMH3","colab_type":"text"},"source":["**How can I control what commands do?**\n","\n","A flag's name usually indicates its purpose (for example, -n is meant to signal \"number of lines\"). Command flags don't have to be a - followed by a single letter, but it's a widely-used convention.\n","\n","- Display the first 5 lines of winter.csv in the seasonal directory."]},{"cell_type":"code","metadata":{"id":"_8UGzxebxVp2","colab_type":"code","colab":{}},"source":["! head -n 5 seasonal/winter.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"--DkJ498xmbI","colab_type":"text"},"source":["**How can I list everything below a directory?**\n","\n","In order to see everything underneath a directory, no matter how deeply nested it is, you can give ls the flag -R (which means \"recursive\"). \n","\n","To help you know what is what, ls has another flag -F that prints a / after the name of every directory and a * after the name of every runnable program. Run ls with the two flags, -R and -F, and the absolute path to your home directory to see everything it contains. (The order of the flags doesn't matter, but the directory name must come last.)"]},{"cell_type":"code","metadata":{"id":"YwGZWrBHyc9L","colab_type":"code","colab":{}},"source":["! ls -R -F ~"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Pi0qeaj4UudG","colab_type":"text"},"source":["**How can I get help for a command?**\n","\n","To find out what commands do, people used to use the man command (short for \"manual\").\n","\n","1. Read the manual page for the tail command to find out what putting a + sign in front of the number used with the -n flag does.\n","2. Use tail with the flag -n +7 to display all but the first six lines of seasonal/spring.csv."]},{"cell_type":"code","metadata":{"id":"Bw8R7MQFU282","colab_type":"code","colab":{}},"source":["! man tail +7"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"99hvXuRzVMV2","colab_type":"code","colab":{}},"source":["! tail -n +7 seasonal/spring.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"sYD55_fsVUwO","colab_type":"text"},"source":["**How can I select columns from a file?**\n","\n","head and tail let you select rows from a text file. If you want to select columns, you can use the command cut. It has several options (use man cut to explore them), but the most common is something like:\n","\n"," cut -f 2-5,8 -d , values.csv\n","\n","\n","which means \"select columns 2 through 5 and columns 8, using comma as the separator\". cut uses -f (meaning \"fields\") to specify columns and -d (meaning \"delimiter\") to specify the separator. You need to specify the latter because some files may use spaces, tabs, or colons to separate columns."]},{"cell_type":"markdown","metadata":{"id":"gxTrNSQzVrFl","colab_type":"text"},"source":["**What can't cut do?**\n","\n","cut is a simple-minded command. In particular, it doesn't understand quoted strings. "]},{"cell_type":"markdown","metadata":{"id":"0INL9lZuV8GU","colab_type":"text"},"source":["**How can I repeat commands?**\n","\n","One of the biggest advantages of using the shell is that it makes it easy for you to do things over again. If you run some commands, you can then press the up-arrow key to cycle back through them. You can also use the left and right arrow keys and the delete key to edit them. Pressing return will then run the modified command.\n","\n","Even better, history will print a list of commands you have run recently. Each one is preceded by a serial number to make it easy to re-run particular commands: just type !55 to re-run the 55th command in your history (if you have that many). You can also re-run a command by typing an exclamation mark followed by the command's name, such as !head or !cut, which will re-run the most recent use of that command."]},{"cell_type":"markdown","metadata":{"id":"63yNfd83WIH0","colab_type":"text"},"source":["1. Run head summer.csv in your home directory (which should fail).\n","2. Change directory to seasonal.\n","3. Re-run the head command with !head.\n","4. Use history to look at what you have done.\n","5. Re-run head again using ! followed by a command number."]},{"cell_type":"code","metadata":{"id":"zh7u_73UVevV","colab_type":"code","colab":{}},"source":["! cd seasonal"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"oyRx_zE9WWtU","colab_type":"code","colab":{}},"source":["! !head"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"n6IDrTMVWfFM","colab_type":"code","colab":{}},"source":["! history"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"NpS4BZQpWjWz","colab_type":"code","colab":{}},"source":["! !5"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ILRw-5v9Wzxr","colab_type":"text"},"source":["**How can I select lines containing specific values?**\n","\n","head and tail select rows, cut selects columns, and grep selects lines according to what they contain. In its simplest form, grep takes a piece of text followed by one or more filenames and prints all of the lines in those files that contain that text. For example, grep bicuspid seasonal/winter.csv prints lines from winter.csv that contain \"bicuspid\".\n","\n","grep can search for patterns as well; we will explore those in the next course. What's more important right now is some of grep's more common flags:\n","\n","- c: print a count of matching lines rather than the lines themselves\n","-h: do not print the names of files when searching multiple files\n","-i: ignore case (e.g., treat \"Regression\" and \"regression\" as matches)\n","-l: print the names of files that contain matches, not the matches\n","-n: print line numbers for matching lines\n","-v: invert the match, i.e., only show lines that don't match"]},{"cell_type":"markdown","metadata":{"id":"WpHxtxwFXAaj","colab_type":"text"},"source":["1. Print the contents of all of the lines containing the word molar in seasonal/autumn.csv by running a single command while in your home directory. Don't use any flags."]},{"cell_type":"code","metadata":{"id":"pX98-Q-jW4G7","colab_type":"code","colab":{}},"source":["! grep molar seasonal/autumn.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5jOrmMEiXDVb","colab_type":"text"},"source":["2. Invert the match to find all of the lines that don't contain the word molar in seasonal/spring.csv, and show their line numbers. Remember, it's considered good style to put all of the flags before other values like filenames or the search term \"molar\"."]},{"cell_type":"code","metadata":{"id":"rQpiYlrwXD4z","colab_type":"code","colab":{}},"source":["! grep -v -n molar seasonal/spring.csv "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"uFPjAbDQXD-j","colab_type":"text"},"source":["3. Count how many lines contain the word incisor in autumn.csv and winter.csv combined. (Again, run a single command from your home directory.)"]},{"cell_type":"code","metadata":{"id":"d2inT9MBYQ96","colab_type":"code","colab":{}},"source":["! grep -c incisor seasonal/autumn.csv seasonal/winter.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"p6zu0SC1YioS","colab_type":"text"},"source":["**Why isn't it always safe to treat data as text?**\n","\n","The SEE ALSO section of the manual page for cut refers to a command called paste that can be used to combine data files instead of cutting them up.\n","\n","Read the manual page for paste, and then run paste to combine the autumn and winter data files in a single table using a comma as a separator. What's wrong with the output from a data analysis point of view?\n","\n","Ans: The last few rows have the wrong number of columns, because joining the lines with columns creates only one empty column at the start, not two."]},{"cell_type":"markdown","metadata":{"id":"XQzyvFIns_ek","colab_type":"text"},"source":["## 3. Combining tools"]},{"cell_type":"markdown","metadata":{"id":"kslx4NoqZIFP","colab_type":"text"},"source":["**How can I store a command's output in a file?**\n","\n","All of the tools you have seen so far let you name input files. Most don't have an option for naming an output file because they don't need one. Instead, you can use redirection to save any command's output anywhere you want. \n","\n","If you run this command:\n","\n","> $ head -n 5 seasonal/summer.csv > top.csv\n","\n","nothing appears on the screen. Instead, head's output is put in a new file called top.csv. The greater-than sign > tells the shell to redirect head's output to a file. It isn't part of the head command; instead, it works with every shell command that produces output.\n","\n","\n","\n","- Combine tail with redirection to save the last 5 lines of seasonal/winter.csv in a file called last.csv."]},{"cell_type":"code","metadata":{"id":"eOfOkPf6tChz","colab_type":"code","colab":{}},"source":["! tail -n 5 seasonal/winter.csv > last.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ZDptZ5wjcRdT","colab_type":"text"},"source":["**How can I use a comman's output as an input?**\n","\n","Suppose you want to get lines from the middle of a file. More specifically, suppose you want to get lines 3-5 from one of our data files. You can start by using head to get the first 5 lines and redirect that to a file, and then use tail to select the last 3:\n","\n","$ head -n 5 seasonal/winter.csv > top.csv\n","\n","$ tail -n 3 top.csv\n","\n","A quick check confirms that this is lines 3-5 of our original file, because it is the last 3 lines of the first 5."]},{"cell_type":"markdown","metadata":{"id":"7vWzb0DrckRy","colab_type":"text"},"source":["1. Select the last two lines from seasonal/winter.csv and save them in a file called bottom.csv.\n","2. Select the first line from bottom.csv in order to get the second-to-last line of the original file."]},{"cell_type":"code","metadata":{"id":"EoRbS8_uchSS","colab_type":"code","colab":{}},"source":["! tail -n 2 seasonal/winter.csv > bottom.csv"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"I2V_Sl-rczDi","colab_type":"code","colab":{}},"source":["! head -n 1 bottom.csv"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"x3itJDqvlzAM","colab_type":"text"},"source":["**What's a better way to combine commands?**\n","\n","Using redirection to combine commands has two drawbacks:\n","\n","1. It leaves a lot of intermediate files lying around (like top.csv).\n","2. The commands to produce your final result are scattered across several lines of history.\n","\n","Instead of sending head's output to a file, add a vertical bar and the tail command without a filename:\n","\n","$ head -n 5 seasonal/summer.csv | tail -n 3\n","\n","The pipe symbol tells the shell to use the output of the command on the left as the input to the command on the right.\n","\n","- Use cut to select all of the tooth names from column 2 of the comma delimited file seasonal/summer.csv, then pipe the result to grep, with an inverted match, to exclude the header line containing the word \"Tooth\". "]},{"cell_type":"code","metadata":{"id":"QzmxteQCn2JL","colab_type":"code","colab":{}},"source":["! cut -d, -f 2 seasonal/summer.csv | grep -v Tooth"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"NuyVPcA7oqo1","colab_type":"text"},"source":["**How can I combine many commands?**\n","\n","You can chain any number of commands together. For example, this command:\n","\n","> $ cut -d , -f 1 seasonal/spring.csv | grep -v Date | head -n 10\n","\n","\n","1. select the first column from the spring data;\n","2. remove the header line containing the word \"Date\"; and\n","3. select the first 10 lines of actual data."]},{"cell_type":"markdown","metadata":{"id":"2G86K0Wtc0lb","colab_type":"text"},"source":["- In the previous exercise, you used the following command to select all the tooth names from column 2 of seasonal/summer.csv:\n","\n","> $ cut -d , -f 2 seasonal/summer.csv | grep -v Tooth\n","\n","\n","Extend this pipeline with a head command to only select the very first tooth name."]},{"cell_type":"code","metadata":{"id":"v2fePtBbmMqH","colab_type":"code","colab":{}},"source":["! cut -d , -f 2 seasonal/summer.csv | grep -v Tooth | head -n 1"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"evQbRLnudZCq","colab_type":"text"},"source":["**How can I count the records in a file?**\n","\n","The command wc (short for \"word count\") prints the number of characters, words, and lines in a file. You can make it print only one of these using -c, -w, or -l respectively.\n","\n","- Count how many records in seasonal/spring.csv have dates in July 2017. To do this, use grep with a partial date to select the lines and pipe this result into wc with an appropriate flag to count the lines."]},{"cell_type":"code","metadata":{"id":"KBxPLKrxdhkJ","colab_type":"code","colab":{}},"source":["! grep 2017-07 seasonal/spring.csv | wc -l"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"uWM2HPF1d_Tq","colab_type":"text"},"source":["**How can I specify many files at once?**\n","\n","Most shell commands will work on multiple files if you give them multiple filenames.\n","\n","To make your life better, the shell allows you to use wildcards to specify a list of files with a single expression. The most common wildcard is * , which means \"match zero or more characters\". Using it, we can shorten the cut command above to this:\n","\n","> $ cut -d , -f 1 seasonal/*\n","or:\n","\n","> $ cut -d , -f 1 seasonal/*.csv"]},{"cell_type":"markdown","metadata":{"id":"KUaJGuZXeNuy","colab_type":"text"},"source":["- Write a single command using head to get the first three lines from both seasonal/spring.csv and seasonal/summer.csv, a total of six lines of data, but not from the autumn or winter data files. Use a wildcard instead of spelling out the files' names in full."]},{"cell_type":"code","metadata":{"id":"myxy1MWIeLLB","colab_type":"code","colab":{}},"source":["! head -n 3 seasonal/s*"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zdql96lGgp16","colab_type":"text"},"source":["**What other wildcards can I use?**\n","\n","The shell has other wildcards as well, though they are less commonly used:\n","\n","- ? matches a single character, so 201?.txt will match 2017.txt or 2018.txt, but not 2017-01.txt.\n","- [...] matches any one of the characters inside the square brackets, so 201[78].txt matches 2017.txt or 2018.txt, but not 2016.txt.\n","- {...} matches any of the comma-separated patterns inside the curly brackets, so {*.txt, *.csv} matches any file whose name ends with .txt or .csv, but not files whose names end with .pdf."]},{"cell_type":"markdown","metadata":{"id":"Nr4CJLJ7hCsy","colab_type":"text"},"source":["**How can I sort lines of text?**\n","\n","As its name suggests, sort puts data in order. By default it does this in ascending alphabetical order, but the flags -n and -r can be used to sort numerically and reverse the order of its output, while -b tells it to ignore leading blanks and -f tells it to fold case (i.e., be case-insensitive). Pipelines often use grep to get rid of unwanted records and then sort to put the remaining records in order.\n","\n","- Remember the combination of cut and grep to select all the tooth names from column 2 of seasonal/summer.csv?\n","\n","> $ cut -d , -f 2 seasonal/summer.csv | grep -v Tooth\n","\n","\n","Starting from this recipe, sort the names of the teeth in seasonal/winter.csv (not summer.csv) in descending alphabetical order. To do this, extend the pipeline with a sort step."]},{"cell_type":"code","metadata":{"id":"ExEFp2dQgv0q","colab_type":"code","colab":{}},"source":["! cut -d , -f 2 seasonal/winter.csv | grep -v Tooth | sort -r"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5K0at0GmhcKr","colab_type":"text"},"source":["**How can I remove duplicate lines?**\n","\n","Another command that is often used with sort is uniq, whose job is to remove duplicated lines. More specifically, it removes *adjacent* duplicated lines. \n","\n","Write a pipeline to:\n","\n","- get the second column from seasonal/winter.csv,\n","- remove the word \"Tooth\" from the output so that only tooth names are displayed,\n","- sort the output so that all occurrences of a particular tooth name are adjacent; and\n","- display each tooth name once along with a count of how often it occurs.\n","\n","\n","The start of your pipeline is the same as the previous exercise:\n","\n","> $ cut -d , -f 2 seasonal/winter.csv | grep -v Tooth\n","\n","Extend it with a sort command, and use uniq -c to display unique lines with a count of how often each occurs rather than using uniq and wc."]},{"cell_type":"code","metadata":{"id":"bwUu8ZWjhrzr","colab_type":"code","colab":{}},"source":["! cut -d , -f 2 seasonal/winter.csv | grep -v Tooth | sort | uniq -c"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"6SlEP5tsh6dK","colab_type":"text"},"source":["**How can I save the output of a pipe?**\n","\n","The shell lets us redirect the output of a sequence of piped commands:\n","\n","> $ cut -d , -f 2 seasonal/*.csv | grep -v Tooth > teeth-only.txt\n","\n","However, > must appear at the end of the pipeline."]},{"cell_type":"markdown","metadata":{"id":"uZx-4XJTiNcK","colab_type":"text"},"source":["**How can I stop a running program?**\n","\n","The commands and scripts that you have run so far have all executed quickly, but some tasks will take minutes, hours, or even days to complete. You may also mistakenly put redirection in the middle of a pipeline, causing it to hang up. If you decide that you don't want a program to keep running, you can type Ctrl + C to end it. This is often written ^C in Unix documentation; note that the 'c' can be lower-case."]},{"cell_type":"markdown","metadata":{"id":"7CFYOqibiU6T","colab_type":"text"},"source":["**Wrapping up**\n","\n","To wrap up, you will build a pipeline to find out how many records are in the shortest of the seasonal data files.\n","\n","1. Use wc with appropriate parameters to list the number of lines in all of the seasonal data files. (Use a wildcard for the filenames instead of typing them all in by hand.)\n","\n","2. Add another command to the previous one using a pipe to remove the line containing the word \"total\".\n","\n","3. Add two more stages to the pipeline that use sort -n and head -n 1 to find the file containing the fewest lines."]},{"cell_type":"code","metadata":{"id":"nzauHpdyk-jK","colab_type":"code","colab":{}},"source":["! wc -l seasonal/*"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"bsSfb8r7ibiy","colab_type":"code","colab":{}},"source":["! wc -l seasonal/*.csv | grep -v total"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Zja_11uSlEH3","colab_type":"code","colab":{}},"source":["! wc -l seasonal/*.csv | grep -v total | sort -n | head -n 1"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"iT6j5kSVtCnC","colab_type":"text"},"source":["## 4. Batch processing"]},{"cell_type":"markdown","metadata":{"id":"4LI2008RlQPP","colab_type":"text"},"source":["**How does the shell store information?**\n","\n","Like other programs, the shell stores information in variables. Some of these, called environment variables, are available all the time. Environment variables' names are conventionally written in upper case, and a few of the more commonly-used ones are shown below.\n","\n",">Variable | Purpose | Value\n",">--- | --- | ---\n",">HOME  | User's home directory | /home/repl\n",">PWD  | Present working directory | Same as pwd command\n",">SHELL | Which shell program is being used | /bin/bash\n",">USER | User's ID | repl\n","\n","To get a complete list (which is quite long), you can type set in the shell.\n","\n"]},{"cell_type":"markdown","metadata":{"id":"t48-Rie0mO2d","colab_type":"text"},"source":["- Use set and grep with a pipe to display the value of HISTFILESIZE, which determines how many old commands are stored in your command history. What is its value?"]},{"cell_type":"code","metadata":{"id":"5rbSpjpRmZXM","colab_type":"code","colab":{}},"source":["! set | grep HISTFILESIZE # 2000"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"tVppQWLgmd2U","colab_type":"text"},"source":["**How can I print a variable's value?**\n","\n","A simpler way to find a variable's value is to use a command called echo, which prints its arguments. Typing\n","\n","> $ echo hello DataCamp!\n","prints\n","\n","> $ hello DataCamp!\n","If you try to use it to print a variable's value like this:\n","\n","> $ echo USER\n","it will print the variable's name, USER.\n","\n","To get the variable's value, you must put a dollar sign $ in front of it. Typing\n","\n","> ! echo $USER\n","\n","prints\n","\n",">  $ repl\n","\n","\n","This is true everywhere: to get the value of a variable called X, you must write $X. (This is so that the shell can tell whether you mean \"a file named X\" or \"the value of a variable named X\".)\n","\n","- The variable OSTYPE holds the name of the kind of operating system you are using. Display its value using echo."]},{"cell_type":"code","metadata":{"id":"txGh1T66tEtC","colab_type":"code","colab":{}},"source":["! echo $OSTYPE"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SO8p61vxnJGT","colab_type":"text"},"source":["**How else does the shell store information?**\n","\n","The other kind of variable is called a shell variable, which is like a local variable in a programming language.\n","\n","To create a shell variable, you simply assign a value to a name:\n","\n","> $ training=seasonal/summer.csv\n","\n","without any spaces before or after the = sign. Once you have done this, you can check the variable's value with:\n","\n","> ! echo $training"]},{"cell_type":"markdown","metadata":{"id":"MSRL1VA3nTmc","colab_type":"text"},"source":["1. Define a variable called testing with the value seasonal/winter.csv.\n","2. Use head -n 1 SOMETHING to get the first line from seasonal/winter.csv using the value of the variable testing instead of the name of the file."]},{"cell_type":"code","metadata":{"id":"Rodfwj0gnQuj","colab_type":"code","colab":{}},"source":["! testing=seasonal/winter.csv"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"OcoTJWLgnbQE","colab_type":"code","colab":{}},"source":["! head -n 1 $testing"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"43AZhzmansdT","colab_type":"text"},"source":["**How can I repeat a command many times?**\n","\n","Shell variables are also used in loops, which repeat commands many times. If we run this command:\n","\n","> ! for filetype in gif jpg png; do echo $filetype; done\n","\n","it produces:\n","\n","> gif\n","> jpg\n","> png\n","\n","Notice these things about the loop:\n","\n","1. The structure is for ...variable... in ...list... ; do ...body... ; done\n","2. The list of things the loop is to process (in our case, the words gif, jpg, and png).\n","3. The variable that keeps track of which thing the loop is currently processing (in our case, filetype).\n","4. The body of the loop that does the processing (in our case, echo $filetype).\n","\n","Notice that the body uses $filetype to get the variable's value instead of just filetype, just like it does with any other shell variable. Also notice where the semi-colons go: the first one comes between the list and the keyword do, and the second comes between the body and the keyword done."]},{"cell_type":"markdown","metadata":{"id":"CyUiykwFn9yr","colab_type":"text"},"source":["- Modify the loop so that it prints:\n","\n","> docx\n","> odt\n","> pdf\n","\n","Please use filetype as the name of the loop variable."]},{"cell_type":"code","metadata":{"id":"Rw52e8xKoC0j","colab_type":"code","colab":{}},"source":["! for filetype in docx odt pdf; do echo $filetype; done"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"fK2Z-cYerBtH","colab_type":"text"},"source":["**How can I repeat a command once for each file?**\n","\n","You can always type in the names of the files you want to process when writing the loop, but it's usually better to use wildcards. Try running this loop in the console:\n","\n","> !  for filename in seasonal/* .csv; do echo $filename; done\n","\n","It prints:\n","\n","> seasonal/autumn.csv\n","\n","> seasonal/spring.csv\n","\n",">seasonal/summer.csv\n","\n","> seasonal/winter.csv\n","\n","\n","because the shell expands seasonal/*.csv to be a list of four filenames before it runs the loop.\n","\n","- Modify the wildcard expression to people/* so that the loop prints the names of the files in the people directory regardless of what suffix they do or don't have. Please use filename as the name of your loop variable."]},{"cell_type":"code","metadata":{"id":"lV4XPs1CrYOu","colab_type":"code","colab":{}},"source":["! for filename in people/*; do echo $filename; done"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"UtRbHLYrrmVW","colab_type":"text"},"source":["**How can I record the names of a set of files?**\n","\n","People often set a variable using a wildcard expression to record a list of filenames. For example, if you define datasets like this:\n","\n","> $ datasets=seasonal/*.csv\n","\n","you can display the files' names later using:\n","\n","> ! for filename in  $datasets;\n","\n","> do echo $filename; done\n","\n","\n","This saves typing and makes errors less likely."]},{"cell_type":"markdown","metadata":{"id":"B24ywDGSshRW","colab_type":"text"},"source":["**A variable's name versus its value**\n","\n","A common mistake is to forget to use $ before the name of a variable. When you do this, the shell uses the name you have typed rather than the value of that variable."]},{"cell_type":"markdown","metadata":{"id":"3XMMN9f7swgq","colab_type":"text"},"source":["**How can I run many commands in a single loop?**\n","\n","Printing filenames is useful for debugging, but the real purpose of loops is to do things with multiple files. This loop prints the second line of each data file:\n","\n","> ! for file in seasonal/* .csv; do head -n 2 $file | tail -n 1; done\n","\n","\n","It has the same structure as the other loops you have already seen: all that's different is that its body is a pipeline of two commands instead of a single command."]},{"cell_type":"markdown","metadata":{"id":"dxRiV3vJs9mq","colab_type":"text"},"source":["- Write a loop that produces the same output as\n","\n","> $ grep -h 2017-07 seasonal/*.csv\n","\n","\n","but uses a loop to process each file separately. Please use file as the name of the loop variable, and remember that the -h flag used above tells grep not to print filenames in the output."]},{"cell_type":"code","metadata":{"id":"5pKZ7yRhsbnu","colab_type":"code","colab":{}},"source":["! for file in seasonal/*.csv; do grep -h 2017-07 $file; done"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"kHiCiALdtgYM","colab_type":"text"},"source":["**Why shouldn't I use spaces in filenames?**\n","\n","It's easy and sensible to give files multi-word names like July 2017.csv when you are using a graphical file explorer. However, this causes problems when you are working in the shell. Instead, you have to quote the files' names so that the shell treats each one as a single parameter"]},{"cell_type":"markdown","metadata":{"id":"msV4cYhotzB1","colab_type":"text"},"source":["**How can I do many things in a single loop?**\n","\n","The loops you have seen so far all have a single command or pipeline in their body, but a loop can contain any number of commands. To tell the shell where one ends and the next begins, you must separate them with semi-colons:\n","\n","> ! for f in seasonal/*.csv;\n","\n","> do echo $f;\n","\n","> head -n 2 $f | tail -n 1;\n","\n","> done\n"]},{"cell_type":"markdown","metadata":{"id":"gYDp_8JHtEyA","colab_type":"text"},"source":["## 5. Creating new tools"]},{"cell_type":"markdown","metadata":{"id":"cBX2xbZTs9rP","colab_type":"text"},"source":["**How can I edit a file?**\n","\n","Unix has a bewildering variety of text editors. For this course, we will use a simple one called Nano. If you type nano filename, it will open filename for editing (or create it if it doesn't already exist). You can move around with the arrow keys, delete characters using backspace, and do other operations with control-key combinations:\n","\n","- Ctrl + K: delete a line.\n","- Ctrl + U: un-delete a line.\n","- Ctrl + O: save the file ('O' stands for 'output').\n","- Ctrl + X: exit the editor."]},{"cell_type":"markdown","metadata":{"id":"RInju0-fuSOU","colab_type":"text"},"source":["**Instructions:**\n","\n","Run nano names.txt to edit a new file in your home directory and enter the following four lines:\n","\n","> Lovelace\n","\n","> Hopper\n","\n","> Johnson\n","\n","> Wilson\n","\n","To save what you have written, type Ctrl + O to write the file out, then Enter to confirm the filename, then Ctrl + X and Enter to exit the editor."]},{"cell_type":"code","metadata":{"id":"GLZSsHXHt_sb","colab_type":"code","colab":{}},"source":["! nano names.txt"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"6CIBUAFluOcL","colab_type":"code","colab":{}},"source":["! Lovelace\n","\n","Hopper\n","\n","Johnson\n","\n","Wilson"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"oo3nOjXtwwxg","colab_type":"text"},"source":["**How can I record what I just did?**\n","\n","When you are doing a complex analysis, you will often want to keep a record of the commands you used. You can do this with the tools you have already seen:\n","\n","1. Run history.\n","2. Pipe its output to tail -n 10 (or however many recent steps you want to save).\n","3. Redirect that to a file called something like figure-5.history."]},{"cell_type":"markdown","metadata":{"id":"WVGYABU7w5Cv","colab_type":"text"},"source":["**Instructions**\n","1. Copy the files seasonal/spring.csv and seasonal/summer.csv to your home directory.\n","2. Use grep with the -h flag (to stop it from printing filenames) and -v Tooth (to select lines that don't match the header line) to select the data records from spring.csv and summer.csv in that order and redirect the output to temp.csv.\n","3. Pipe history into tail -n 3 and redirect the output to steps.txt to save the last three commands in a file. (You need to save three instead of just two because the history command itself will be in the list.)"]},{"cell_type":"code","metadata":{"id":"XO3MXeVMtGug","colab_type":"code","colab":{}},"source":["! cp seasonal/spring.csv seasonal/summer.csv ~"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"B5p6EESMxFeo","colab_type":"code","colab":{}},"source":["! grep -h -v Tooth spring.csv summer.csv > temp.csv"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"2b2F79YtxS8H","colab_type":"code","colab":{}},"source":["! history | tail -n 3 > steps.txt"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"99YWgfAexsDX","colab_type":"text"},"source":["**How can I save commands to re-run later?**\n","\n","You have been using the shell interactively so far. But since the commands you type in are just text, you can store them in files for the shell to run over and over again. To start exploring this powerful capability, put the following command in a file called headers.sh:\n","\n","> $ head -n 1 seasonal/*.csv\n","\n","\n","This command selects the first row from each of the CSV files in the seasonal directory. Once you have created this file, you can run it by typing:\n","\n","> $ bash headers.sh\n","\n","\n","This tells the shell (which is just a program called bash) to run the commands contained in the file headers.sh, which produces the same output as running the commands directly."]},{"cell_type":"markdown","metadata":{"id":"xeZLMkohzH6O","colab_type":"text"},"source":["**Instructions:**\n","\n","1. Use nano dates.sh to create a file called dates.sh that contains this command:\n","\n","> $ cut -d , -f 1 seasonal/*.csv\n","\n","to extract the first column from all of the CSV files in seasonal.\n","\n","2. "]},{"cell_type":"code","metadata":{"id":"sKz_Jgcvxybw","colab_type":"code","colab":{}},"source":["! nano dates.sh"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ReyA32V00bit","colab_type":"code","colab":{}},"source":["! cut -d , -f 1 seasonal/*.csv"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"87r02B-l0k_z","colab_type":"code","colab":{}},"source":["! bash dates.sh"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"xm9CjBXX0n_7","colab_type":"text"},"source":["**How can I re-use pipes?**\n","\n","A file full of shell commands is called a * shell script, or sometimes just a \"script\" for short. Scripts don't have to have names ending in .sh, but this lesson will use that convention to help you keep track of which files are scripts.\n","\n","Scripts can also contain pipes. For example, if all-dates.sh contains this line:\n","\n","> $ cut -d , -f 1 seasonal/*.csv | grep -v Date | sort | uniq\n","\n","then:\n","\n","> $ bash all-dates.sh > dates.out\n","\n","will extract the unique dates from the seasonal data files and save them in dates.out."]},{"cell_type":"markdown","metadata":{"id":"Wtk9ye7y07bT","colab_type":"text"},"source":["**Instructions:**\n","\n","1. A file teeth.sh in your home directory has been prepared for you, but contains some blanks. Use Nano to edit the file and replace the two _ _ _ _ placeholders with seasonal/* .csv and -c so that this script prints a count of the number of times each tooth name appears in the CSV files in the seasonal directory."]},{"cell_type":"code","metadata":{"id":"P1vXvpfI05xM","colab_type":"code","colab":{}},"source":["! nano teeth.sh"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"pDw1Wuzy1qck","colab_type":"code","colab":{}},"source":["! cut -d , -f 2 seasonal/*.csv | grep -v Tooth | sort | uniq -c"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"YvIOL-xv3gjA","colab_type":"code","colab":{}},"source":["! bash teeth.sh > teeth.out"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"iG3IqdvCCfZJ","colab_type":"code","colab":{}},"source":["! cat teeth.out"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"e2DqJt_X4pbH","colab_type":"text"},"source":["**How can I pass filenames to scripts?**\n","\n","A script that processes specific files is useful as a record of what you did, but one that allows you to process any files you want is more useful. To support this, you can use the special expression $@ (dollar sign immediately followed by at-sign) to mean \"all of the command-line parameters given to the script\". For example, if unique-lines.sh contains this:\n","\n","> ! sort $@ | uniq\n","\n","then when you run:\n","\n","> ! bash unique-lines.sh seasonal/summer.csv\n","\n","the shell replaces $@ with seasonal/summer.csv and processes one file. If you run this:\n","\n","> ! bash unique-lines.sh seasonal/summer.csv seasonal/autumn.csv\n","\n","it processes two data files, and so on."]},{"cell_type":"markdown","metadata":{"id":"pHvV3rkCFxcB","colab_type":"text"},"source":["1. Edit the script count-records.sh with Nano and fill in the two _ _ _ _ placeholders with $@ and -l respectively so that it counts the number of lines in one or more files, excluding the first line of each."]},{"cell_type":"code","metadata":{"id":"L13J1H4UC33-","colab_type":"code","colab":{}},"source":["! nano count-records.sh"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"nrZsnifrDZ-G","colab_type":"code","colab":{}},"source":["! tail -1 -n +2 $@ | wc -1"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"YTX9vw-JFvCa","colab_type":"text"},"source":["2. Run count-records.sh on seasonal/*.csv and redirect the output to num-records.out using >."]},{"cell_type":"code","metadata":{"id":"uyFEy3TQFqgK","colab_type":"code","colab":{}},"source":["! bash count-records.sh seasonal/*.csv > num-records.out "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SSIrYTu0D4BG","colab_type":"text"},"source":["**How can one shell script do many things?**\n","\n","Our shells scripts so far have had a single command or pipe, but a script can contain many lines of commands. For example, you can create one that tells you how many records are in the shortest and longest of your data files, i.e., the range of your datasets' lengths.\n","\n","Note that in Nano, \"copy and paste\" is achieved by navigating to the line you want to copy, pressing CTRL + K to cut the line, then CTRL + U twice to paste two copies of it."]},{"cell_type":"code","metadata":{"id":"Pt68bvcXD7He","colab_type":"code","colab":{}},"source":["! nano range.sh"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ts_J4NHIs_Gu","colab_type":"code","colab":{}},"source":["! wc -1 $@ | grep -v total"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"XmS_YaSEtvvu","colab_type":"code","colab":{}},"source":["! nano range.sh"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"sxMHRQv1tkVg","colab_type":"code","colab":{}},"source":["! wc -1 $@ | grep -v total | sort -n | head -n 1 "],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"T_KZ4qYduATy","colab_type":"code","colab":{}},"source":["! nano range.sh"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"mpdWG9NfuTaZ","colab_type":"code","colab":{}},"source":["! wc -1 $@ | grep -v total | sort -n -r | head -n 1 "],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"QHgBPKMYuZWw","colab_type":"code","colab":{}},"source":["! bash range.sh seasonal/*.csv > range.out "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"eH60nVTRugkM","colab_type":"text"},"source":["**How can I write loops in a shell script?**\n","\n","Shell scripts can also contain loops. You can write them using semi-colons, or split them across lines without semi-colons to make them more readable:\n","\n","-  Print the first and last data records of each file.\n","> for filename in $ @\n","\n","> do\n",">>    head -n 2 $ filename | tail -n 1\n"," \n"," >> tail -n 1 $f ilename\n"," \n",">done\n","\n","(You don't have to indent the commands inside the loop, but doing so makes things clearer.)\n","\n","The first line of this script is a comment to tell readers what the script does. Comments start with the # character and run to the end of the line. Your future self will thank you for adding brief explanations like the one shown here to every script you write."]},{"cell_type":"markdown","metadata":{"id":"vE724d_wu3DC","colab_type":"text"},"source":["**Instructions**\n","\n","1. Fill in the placeholders in the script date-range.sh with $filename (twice), head, and tail so that it prints the first and last date from one or more files."]},{"cell_type":"code","metadata":{"id":"VOaXoEjyuxnD","colab_type":"code","colab":{}},"source":["! nano date-range.sh"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"USxbBdLMvLp4","colab_type":"code","colab":{}},"source":["for filename in $@\n","do\n","  cut -d , -f 1 $filename | grep -v Date | sort | head -n 1\n","  cut -d , -f l $filename | grep -v Date | sort | tail -n 1\n","done  "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"oJJyPmLNwvWI","colab_type":"text"},"source":["2. Run date-range.sh on all four of the seasonal data files using seasonal/*.csv to match their names."]},{"cell_type":"code","metadata":{"id":"LR4hukNRwwIm","colab_type":"code","colab":{}},"source":["! bash date-range.sh seasonal/*.csv "],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hN1wXZVPw45n","colab_type":"text"},"source":["3. Run date-range.sh on all four of the seasonal data files using seasonal/*.csv to match their names, and pipe its output to sort to see that your scripts can be used just like Unix's built-in commands."]},{"cell_type":"code","metadata":{"id":"ZyFygx3iw5kl","colab_type":"code","colab":{}},"source":["! bash date-range.sh seasonal/*.csv | sort"],"execution_count":0,"outputs":[]}]}