Mitchells shell tips

.bashrc additions or shell configuration

For easy project sharing add this to your .bashrc

umask 002

Using SMRTlink tools

PATH=$PATH:/gscratch/stergachislab/install_dir/smrtlink/smrtcmds/bin/

More detailed squeue results

 alias myq=$'squeue -u $(whoami) -o $\'%A\t%u\t%C\t%m\t%a\t%M\t%p\t%T\t%r\t%j\t%N\' --noconvert | column -t'

Better history with the up arrow in shell

For better history text completion in bash add the following to ~/.inputrc:

# Press up-arrow for previous matching command
"\e[A":history-search-backward
# Press down-arrow for next matching command
"\e[B":history-search-forward

see for details: https://unix.stackexchange.com/questions/96510/search-for-a-previous-command-with-the-same-prefix-when-i-press-up-at-a-shell-pr

bash scripting

Start every bash script with

These options make it so that once there is an error in the bash script the script ends with an error status. It also errors if there are unset variables (-u).

#!/usr/bin/env bash                                                                                                                                                                                            
set -euo pipefail 

ssh tips

Want to only ssh in once per terminal, add this to a file called ~/.ssh/config to set up something called a controlmaster .

Host stergachis
  HostName klone.hyak.uw.edu    
  User mvollger
  controlmaster auto
  controlpath /tmp/ssh-ster-%r@h:%p
  ServerAliveInterval 60
  ForwardAgent yes 

modify your user name, and then type:

ssh stergachis

and then all subsequent calls to

ssh stergachis

won’t require that you login as long as the first session is still open.

Don't copy data, use softlinks

Lets say there are some subreads in /mmfs1/gscratch/stergachislab/data/... that you want to use in your project. Don't copy these files, instead use softlinks. e.g.

ln -s /mmfs1/gscratch/stergachislab/data/...subreads.bam /path/to/my/project/data/.

This puts a link to the file in a convenient place for you without copying all the data, and you can use these links like regular files with all your programs 😄 !

Some Unix tools I love

• Starship is a minimal, blazing fast, and extremely customizable prompt for any shell! Starship: Cross-Shell Prompt
• lsd, alias ls to lsd for a pretty terminal
• fd an simpler/faster version of find
• ripgrep/rg a faster version of grep
• A whole lot more

See how many jobs you can use on checkpoint

first add this to your .bashrc

alias joblimit=$'squeue  | grep -w R | grep ckpt |  wc -l | awk \'{print "Running Jobs: "$1}\' && sacctmgr show assoc where account=ckpt format=GrpJobs | grep -P "\d+" | awk \'{print "Allowed Jobs: "$1}\''

then

$ joblimit
Running Jobs: 65
Allowed Jobs: 64

Making and working with compressed bed files

Use compression with your bed files! The files will work seamlessly with bedtools, pandas, fread, etc and often compressing and writing is faster than just writing because you are writing a smaller file and writes are very slow.

To compress use the tool bgzip which is part of htslib and bcftools. It is compatible with all gzip tools but it allows for multithreading and indexing. e.g.:

bgzip -@ 8 fake.bed

will make fake.bed.gz and it will be much smaller. It is actually better to never even make the uncompressed version, an example with fibertools:

ft extract fake.bam --m6a - | bgzip -@ 8 > m6a.bed.gz

Furthermore if you make a compressed file with bgzip you can index it using tabix which can really speed up analysis. e.g.:

tabix -p bed m6a.bed.gz

creates the file m6a.bed.gz.tbi and then you can do things like

tabix m6a.bed.gz chr20:20000-40000

to extract only a specific region you are interested in!

This allow works with fasta files e.g.:

bgzip -@ 8 hg38.fa

results in hg38.fa.gz which you can index with:

samtools faidx hg38.fa.gz

and then get any region you want with:

samtools faidx hg38.fa.gz chr20:20000-40000