5. Bash commands

The codeblock below describes a list of common bash commands used by the lab:

## print working directory
pwd

## change working directory
cd /path/to/directory

### the path supplied can be absolute or relative.
### example: 
### your working directory is /media/cdn-bc/RAID/ and you wish to cd to the Datasets directory.
### you can:
cd /media/cdn-bc/RAID/Datasets    # using absolute path or
cd Datasets                       # using relative path

### We can use special characters to replace paths:
cd ..    # move up a directory
cd ~     # move to home directory

## list files in directories
ls     # simplest way of listing files
ls -l  # the -l flag will list more information on the files and directories
ls -a  # list all files including hidden files
ls -al # multiple flags can be used at once
ls -lh # list all file info and format the file size in a human-readable format

### You may also perform `ls` of other directories by supplying the path to that directory 

## create new directory
mkdir dir1   # mkdir is short for make directory
mkdir dir1 dir2 dir3 dir4  # multiple directories can be created at once
mkdir -p dir1   # Fursham likes the `-p` flag, this will not return errors when `dir1` already exists

## create new text files
touch file1.txt  # creates empty text file
nano file1.txt   # creates text file and opens it up in nano for editing
echo "THIS IS MY TEXT" > file1.txt   # create a text file with the text in quotations

## deleting files and directories
### WARN: this permanently deletes files and directories
rm file1
rm -r dir1   # always use `-r` for directories. `-r` is short for recursive 

## moving files and directories
mv file1 /path/to/destination/
mv -r dir1 /path/to/destination/

## copying files and directories
cp file1 /path/to/destination/
cp -r dir1 /path/to/destination/

When working with large text files, it is useful to know simple commands to preview, and summarise its content. The codeblock below shares some commands to do so:

## compressing and decompressing files
## this depends on the compression type
### .zip files:
zip zipfilename.zip file1
zip zipfilename.zip file1 file2     # multiple files can be zipped
zip -r zipfilename.zip dir1 file1   # activate `-r` flag for zipping directories
unzip zipfilename.zip  #unzipping

### .gz files
gzip file1
gzip -k file1   # input files will be kept
gunzip file.gz

### .tar files
tar -cvf files.tar.gz file1 file2 file3   # create a .tar file
tar -czvf files.tar.gz file1 file2 file3   # create a .tar.gz file
tar -cjvf files.tar.gz file1 file2 file3   # create a .tar.bzip file
tar -xvf files.tar.gz   # decompress a tar archive
tar -zxvf files.tar.gz   # decompress a tar.gz archive
tar -jxvf files.tar.bzip   # decompress a tar.bzip archive

## previewing files
head file1
head -n 20 file1    #preview first 20 lines
cat file1    # prints entire content of file
less file1

## previewing gzip-compressed files
zcat file1.gz
zcat file1.gz | head  # `|` is a pipe operator which pipes the output of zcat to the next function
zcat file1.gz | less
gunzip -c file1 # I prefer zcat to this

## counting characters/lines in files
wc -l file1  # counting number of lines
wc -w file1  # counting number of words
wc -m file1  # counting number of characters
zcat file1.gz | wc -l # counting number of lines of .gz file

## regular expression and pattern matching
### very useful to quickly check instances of pattern in files
grep "word" file1   # returns lines from file1 with "word" 
grep -v "word" file1  # returns lines from file1 that do not contain "word"
grep "^word" file1   # returns lines from file 1 that begins with "word"
grep -A1 "word" file1  # returns 1 leading line together with matching lines
grep -B1 "word" file1   # returns 1 trailing line together with matching lines

Below are some common commands to create and modify variables in BASH

## create new variables
VAR="SOME TEXT"

## save command output to variables
VAR=$( pwd )

## get basename of path that is stored in variables
VAR=$( pwd )
VAR=$( basename $VAR )

## remove prefix/suffixes from character variables
VAR=01_somefile.txt
VAR=${VAR%.txt}   # remove .txt suffix
VAR=${VAR#01_}    # remove 01_ prefix

## replace strings from character variables
VAR=01_somefile.txt
VAR=${VAR/.txt/.tsv}   # replace txt to tsv

Fursham's rule of thumb:
If you need to write the code more than twice, you need to automate it!

So, create a while or for loop to perform the same function(s) on multiple files:

## perform while loop using tsv files as input
while IFS=$'\t' read -r COL1 COL2;do
    # function
done < /PATH/TO/TEXTFILE.tsv

## perform while loop using csv files as input
while IFS=$',' read -r COL1 COL2;do
    # function
done < /PATH/TO/TEXTFILE.csv

## perform for loops
for file in *;do    # iterates all files in current directory
  # function. filenames are stored in $file variable
done

# to be strict with the files for iteration, we can replace * with other patterns:
#  *.tsv   (only tsv files)
#  01_*    (only files that begin with 01)

These are the aliases that are currently coded in Clarke

..    # same as cd ..
...   # same as cd ../..
RAID  # set wd to the RAID drive
lss   # same as less -S
ldir  # list out directories in wd
htopme  # htop of your processes