Skip to content

Mitchells shell tips

Jump to bottom
Mitchell Robert Vollger edited this page Aug 20, 2024 · 7 revisions

.bashrc additions or shell configuration

For easy project sharing add this to your .bashrc 

umask 002

Add common tools for the stergachis lab

export PATH=$PATH:/mmfs1/gscratch/stergachislab/bin

(this will five you fibertools aka ft among others)

Using SMRTlink tools

PATH=$PATH:/gscratch/stergachislab/install_dir/smrtlink/smrtcmds/bin/

More detailed squeue results

 alias myq=$'squeue -u $(whoami) -o $\'%A\t%u\t%C\t%m\t%a\t%M\t%p\t%T\t%r\t%j\t%N\' --noconvert | column -t'

Better history with the up arrow in shell

For better history text completion in bash add the following to ~/.inputrc:

# Press up-arrow for previous matching command
"\e[A":history-search-backward
# Press down-arrow for next matching command
"\e[B":history-search-forward

see for details: https://unix.stackexchange.com/questions/96510/search-for-a-previous-command-with-the-same-prefix-when-i-press-up-at-a-shell-pr

bash scripting

Start every bash script with

These options make it so that once there is an error in the bash script the script ends with an error status. It also errors if there are unset variables (-u).

#!/usr/bin/env bash                                                                                                                                                                                            
set -euo pipefail

ssh tips

Want to only ssh in once per terminal, add this to a file called ~/.ssh/config to set up something called a controlmaster .

Host stergachis
  HostName klone.hyak.uw.edu    
  User mvollger
  controlmaster auto
  controlpath /tmp/ssh-ster-%r@h:%p
  ServerAliveInterval 60
  ForwardAgent yes

modify your user name, and then type:

ssh stergachis

and then all subsequent calls to

ssh stergachis

won’t require that you login as long as the first session is still open.

Don't copy data, use softlinks

Lets say there are some subreads in /mmfs1/gscratch/stergachislab/data/... that you want to use in your project. Don't copy these files, instead use softlinks. e.g.

ln -s /mmfs1/gscratch/stergachislab/data/...subreads.bam /path/to/my/project/data/.

This puts a link to the file in a convenient place for you without copying all the data, and you can use these links like regular files with all your programs 😄 !

Some Unix tools I love

  • Starship is a minimal, blazing fast, and extremely customizable prompt for any shell! Starship: Cross-Shell Prompt
  • lsd, alias ls to lsd for a pretty terminal
  • fd an simpler/faster version of find
  • ripgrep/rg a faster version of grep
  • A whole lot more

See how many jobs you can use on checkpoint

first add this to your .bashrc

alias joblimit=$'squeue  | grep -w R | grep ckpt |  wc -l | awk \'{print "Running Jobs: "$1}\' && sacctmgr show assoc where account=ckpt format=GrpJobs | grep -P "\d+" | awk \'{print "Allowed Jobs: "$1}\''

then

$ joblimit
Running Jobs: 65
Allowed Jobs: 64

Making and working with compressed bed files

Use compression with your bed files! The files will work seamlessly with bedtools, pandas, fread, etc and often compressing and writing is faster than just writing because you are writing a smaller file and writes are very slow.

To compress use the tool bgzip which is part of htslib and bcftools. It is compatible with all gzip tools but it allows for multithreading and indexing. e.g.:

bgzip -@ 8 fake.bed

will make fake.bed.gz and it will be much smaller. It is actually better to never even make the uncompressed version, an example with fibertools:

ft extract fake.bam --m6a - | bgzip -@ 8 > m6a.bed.gz

Furthermore if you make a compressed file with bgzip you can index it using tabix which can really speed up analysis. e.g.:

tabix -p bed m6a.bed.gz

creates the file m6a.bed.gz.tbi and then you can do things like

tabix m6a.bed.gz chr20:20000-40000

to extract only a specific region you are interested in!

This aslo works with fasta files e.g.:

bgzip -@ 8 hg38.fa

results in hg38.fa.gz which you can index with:

samtools faidx hg38.fa.gz

and then get any region you want with:

samtools faidx hg38.fa.gz chr20:20000-40000

Simplify using tmux

I had a conversation about how tmux is hard to remember to use, and I generally agree. So, this tip is for people who are struggling to use it consistently.

So, I make it easy for myself in this way. I have this script in my home directory called start.sh, and every time I log into a node for the first time, I run this script.

#!/bin/sh

startDir=$HOME/
S='rand'

# create session with a window called 1 in a detached state
tmux new -s $S -n one -d

# select the window I want to run stuff in , session S window 0
tmux select-window -t "$S":1
# I can them exacute a command in that session like this, C-m acts like a carrige return
tmux send-keys -t $S:1 "${cmd}" C-m "cd $startDir" C-m

# add new windos and do the same
tmux new-window  -t "$S":2 -n two
tmux send-keys -t $S:2 "${cmd}" C-m "cd $startDir" C-m

# add new windos and do the same
tmux new-window  -t "$S":3 -n three
tmux send-keys -t $S:3 "${cmd}" C-m "cd $startDir" C-m




tmux select-window -t "$S":0
# attach the tmux session
tmux attach -t $S

And it sets up a tmux session with three windows on that node called rand

I then have this alias set in my .bashrc so that if I every lose connection I can always reattach my session by just typing rand

alias rand="tmux detach-client -s rand; tmux attach -t rand"

For me having these shortcuts made it so I use tmux 95% of the time instead of 5%. Hope it helps you too!

Clone this wiki locally