-
Notifications
You must be signed in to change notification settings - Fork 0
Unix and the command line
A few quick notes: many of these commands can use certain special characters to do certain things. Most will let you use "wildcards" to match multiple files or directories at once. Use * to match any number of any character; use ? to match a single character of any time. The output of a given command can be "piped" into the next command using a pipe character - | - and can be directed to an output file of your choice using the right angle bracket - >. You can chain together unrelated commands using && or ; (but for the first, if an earlier command fails, the rest will not run.) Finally, ~ signifies your home directory location. Note that if you are used to including spaces in filenames and folders, you will always need to include quotations around them ("\your\badly named\directory" and "Bad filename.txt"), because otherwise ~every command you use will break horribly. Life will get much easier if you just use underscores (_) instead. Similarly, consider including the date at the beginning of every filename in YYYYMMDD format, because your files will always show up in a predictable order.
Get your full current directory ("present working directory")
% pwd
List files in your current directory. Add -la if you want extra info (dates, size, permissions.) The last example shows you detailed information about any file whose name ends in .txt.
% ls
% ls -la
% ls -la *.txt
Change directory. In these examples, you're switching to a subdirectory named dirname in your current directory, to the directory above your current directory, and to a subdirectory in that directory called subdir, switching to your home directory, and switching to a subdirectory called otherdir in your home directory.
% cd dirname
% cd ..
% cd ../subdir
% cd ~
% cd ~/otherdir
Make directory. In these examples, you are making a subdirectory named dirname in your current directory, making a subdirectory named subdirin th e directory above your current directory, and making a subdirectory called otherdir in your home directory.
% mkdir dirname
% mkdir ../subdir
% mkdir ~/otherdir
Quickly view a text file. Press space to scroll down, and Ctl-C (yes, on Macs too) to quit viewing a big file. For editing files in the terminal, emacs and vim are common choices (but have a steep learning curve.) In these examples, you are looking at a file in your current directory (filename.txt) and one in your homedirectory (otherfile.txt).
% more filename.txt
% more ~/otherfile.txt
Quickly view a selected portion of a text file. These commands show the top (head) or bottom (tail) 15 lines of a given file (here filename.txt). The number can be changed.
% head -n 15 filename.txt
% tail -n 15 filename.txt
Move a file. These examples effectively rename it, rename it and move it to the subdirectory dirname (note that you don't have to rename it - you can use the old name if it doesn't already exist in the new location), rename it and move it to a subdirectory (subdir) in the directory above the current one, and rename it and move it to your home directory. The last command moves any file whose name ends in .txt into the subdirectory dirname.
% mv filename.txt filename2.txt
% mv filename.txt dirname/filename2.txt
% mv filename.txt ../subdir/filename2.txt
% mv filename.txt ~/filename2.txt
% mv *.txt dirname
Copy a file. Unlike mv, you'll still have the original.
% cp filename.txt filename2.txt
% cp filename.txt dirname/filename2.txt
% cp filename.txt ../subdir/filename2.txt
% cp filename.txt ~/filename2.txt
% cp *.txt dirname
Combine text files into one large textfile ("concatenate"). In this example, we take filename.txt and filename2.txt and combine them into one bigger file, the aptly named biggerfile.txt. If we did not include > biggerfile.txt, the concatenated file contents would simply be displayed on the screen.
% cat filename.txt filename2.txt > biggerfile.txt
Delete files. Adding the -f option forces deletion through warnings; adding the -r option makes the command recursive and deletes not just all matching files but all matching subdirectories and files in those subdirectories too. (USE WITH CAUTION! rm -rf is forever!) In this examples, we delete filename.txt (a single file in the current directory), all files in the current directory whose names end in .txt, and in the final example, the subdirectory dirname and all its contents.
% rm filename.txt
% rm *.txt
% rm -rf dirname
You can direct the results from text-producing commands into a text file using the right angle bracket (>). This takes all the GenBank files in a directory, looks for the LOCUS line in them, and puts a list of them the results in a text file:
% grep "LOCUS" *.gbk > gbk-list.txt
You can pass things on to a following command too, using the pipe (|). This looks for the same thing, but only shows you the top 5:
% grep "LOCUS" *.gbk | head -n 5
Open a file in the terminal-friendly text editor emacs (it will make a file if one does not exist). To save, Ctl-x, Ctl-s; to exit, Ctl-x, Ctl-c (my mnemonic: execute save, execute close). (Yes yes you can use vim, but are you actually gonna remember :wq!, let along whether or not you are in a mode that requires you to go for Esc first?)
% emacs filename.txt
Securely copy a file from the server to your computer (NOTE: these commands are run locally on your computer. In MacOS, you can do this via Terminal; in Windows, you can use the Windows Subsystem for Linux (in which case that is a legit Linux install and will run scp). These commands respectively copy file.txt from /your/rc/directory on the RC server to /your/computer/directory on your computer; recursively copy /your/rc/directory and all its contents to /your/computer/directory on your computer; and copy all files ending in .txt to the directory on your computer that you are running t he command in. You can put the local location first to upload things to remote directories, as in the remaining example. Note that you will get asked for your RC password and authentication key every time you run the scp command, so if you have a lot of files, consider either making one compressed file or recursively copying a bunch of files in one command.
% scp jharvard@login.rc.fas.harvard.edu:/your/rc/directory/file.txt /your/computer/directory/file.txt
% scp -r jharvard@login.rc.fas.harvard.edu:/your/rc/directory/ /your/computer/directory/
% scp jharvard@login.rc.fas.harvard.edu:"/your/rc/directory/*.txt" .
% scp /your/computer/directory/file.txt jharvard@login.rc.fas.harvard.edu:/your/rc/directory/file.txt
You may want to start a "session" that you can reload if, say, your WiFi connection drops and your ssh connection gets closed. That normally kills any commands/processes that you had running, but with sessioning tools, you can reload everything. To do this, you have to note what login server you are on - if disconnected, you must log back in to the same server (the session info is server specific):
% uname -a
Linux holylogin02.rc.fas.harvard.edu 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
% ssh jharvard@holylogin02.rc.fas.harvard.edu
Your first option is to start a "session" with screen The first option establishes the session. You can "detach" the session and return to the unadorned login prompt via the second command or Ctl-A, D. When you're logged back in, you can check on what sessions you had using the third command. When you want to "re-attach" a session, you use the fourth command, and from within a session, you can end it for good using the last command.
% screen -S myProject
% screen -d # note that Ctl-A, D does this too
% screen -ls
% screen -r myProject
% exit # when logged into a screen session only!
You can also do ssh-persistent sessions with tmux. The first option establishes the session. You can "detach" the session and return to the unadorned login prompt via the second command or Ctl-B, D. When you want to "re-attach" a session, you use the third command, and from within a session, you can end it for good using the last command. Note that unlike screen, tmux adds a handy info bar at the bottom of the screen that displays info about your session, making it easy to keep track of what's going on.
% tmux new -s myProject
% tmux a -t myProject
% exit # when logged into a screen session only!
Getting help. Most commands will provide a short list of options if you follow the command with --help or sometimes -h (as in the first two examples, which will yield options for cp and cat); man will provide you with more detail (examples here are again for cp and cat.) Frankly, Google will get you more helpful results much of the time.
% cp --help
% cat --help
% man cp
% man cat
If you want to save shortcut versions of commands, you can add them to your .bashrc file (in your homedirectory) as an alias. A few examples are below (note that some would have be be altered for use on macOS or for use in a different shell like zsh - there you alter your .zshrc file.) These aliases respectively provide a shortcut command to head to your scratch directory, shortcuts to activate and deactivate a conda environment, a shortcut to use a lengthy version of the ls command that sorts files by filesize and gives them more human-readable sizes, and a shortcut to use a version of ls that sorts things by how recently they were modified. Once you've edited your .bashrc file, you'll need to reload it (% source ~/.bashrc).
alias scratch='cd /n/holyscratch01/your_lab/jharvard'
alias ena_on='source activate ena_env'
alias ena_off='source deactivate ena_env'
alias lsz='ls --human-readable --size -1 -S --classify'
alias lst='ls -t -1'
Finally, one important note about most of the scripts: these are shell scripts that are essentially big, annotated sets of terminal commands. They're written for Bash, and while they have some fancier components (if/then commands, for and while loops), they're still basically just a big bunch of terminal commands, generally executed after defining a few variables (directory locations and so on) at the beginning. When troubleshooting things, you can often run components more or less line-by line.
You can think of grep as a filter that will look for simple text or more complicated regular expressions in text files - many files, if desired - without you having to open them all up. The first one lists every locus tag for every GenBank file in a directory; the second will show every ferridoxin or flavodoxin hit in a specific GenBank file.
% grep "locus_tag" *.gbk
% grep "f.*doxin" genome.gbk
A detailed sorted list with info about filesizes for a given filetype (here a GenBank (.gb) file), with the sizes listed in a human-readable form:
% ls -lah *.gb
A quick (unsorted) version of ls that hides .txt files with a size of 0:
% ls -1f *.txt | awk '{if ($5 != 0) print $9}'
A version of the same that tells you the number of .txt files with a size > 0 that you have:
% ls -1f *.txt | awk '{if ($5 != 0) print $9}' | wc -l
Have a ridiculous of output files and need to quantify them without crashing ls (!)? Change file suffix as appropriate and get your count here:
% find . -maxdepth 1 -name '*.txt' | wc -l
List the number of subdirectories in your current directory? Use find yet again:
% find . -maxdepth 0 -type d | wc -l
A way to delete .txt files with a size of < 50 bytes in a directory:
% find . -name "*.txt" -type 'f' -size -50c -delete
You can think of this tool as a filter - one you can use to find-and-replace text. You can do this in one or more files without opening them.
% sed -i '' "s/ID/locus_tag/g" genome.gbk
% sed -i '' "s/ID/locus_tag/g" *.gbk
Remove blankspace from the ends of lines with an in-place sed edit - note that this uses "regular expressions" ("regex") to handle things more complicated than searching for a literal string of text (here it targets any number of blank spaces followed by the end of a line):
% sed -i.bak 's/[[:blank:]]*$//' file.sh > file.sh
Like grep and sed, you can think of this as a filter that you can use to manipulate data from a text file. It's... not all that easy to summarize, but it can be pretty handy. E.g. this looks in a tab-delineated textfile for the second column of a file (here a BLAST results file) and returns that data as a sorted list.
% awk -F '\t' '{print $2}' blastResults.txt | sort -u > blastSeqID.txt
You can use loops at the command line. This finds all GenBank files in a directory, grabs the accession and the first locus tag (to get the prefix), and outputs that info into a single final file.
% ls *.gbk > list.txt; while read line; do grep "ACCESSION" $line > acc-temp.txt; grep -m 1 "locus_tag" "$line" | head -1 > loc-temp.txt; cat acc-temp.txt loc-temp.txt >> acc-loc.txt; done < list.txt
% ls *.gbk > list.txt
% while read line; do
grep "ACCESSION" $line > acc-temp.txt;
grep -m 1 "locus_tag" "$line" | head -1 > loc-temp.txt;
cat acc-temp.txt loc-temp.txt >> acc-loc.txt;
done < list.txt
This gets the total number of sequences in all fasta files in a directory
% for i in *; do grep -v ">" "$i" | wc | awk '{print $3-$1}'; done`
You can find plenty of additional dumb bash tricks in the scripts, or with the help of Google & stackoverflow.
- the missing semester of your CS education - Getting comfortable with the command line, version control, text editors, etc. can save so much time for actual computational work and data analysis!
- happy belly bioinformatics - Approachable guides to Unix and R. More on it in a publication.
- the carpentries - Science-focused organization for teaching coding and data analysis. Though built around formal workshops, the curricula can be pretty helpful too.