Skip to content
Permalink
Branch: master
Find file Copy path
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
312 lines (226 sloc) 6.44 KB

2.Linux

1) Examples

  • grep
grep -v -c pattern FILES  #(-v exclude -c count lines)
grep -l pattern FILES #(-l: list of the files)
  • head, tail V.S. cut
  • cat V.S. paste
head -100 FILE
tail -100 FILE
cut -f 2 FILE #(-f: field)
cut -d ; -f 2 FILE #(-d change delimiter)

cat FILES* > NEW_FILE
paste FILES* > NEW_FILE
  • rev and tac
echo john temp | rev     #(output pmet nhoj)
tac    #(reverse by lines)
  • wc
ls * |wc -w   or  wc -l  #(count the number of files)
  • comm and vimdiff, diff
comm -2 file1 file2
diff file1 file2
  • sort
sort at position 1, delimeter is ','; -r reverse sort
sort -t ',' -k 1 sort -r -t ',' -k 1
  • uniq
cat human.gtf | cut –f 2,3 | sort | uniq –c
  • seq
for i in `seq 1 2 10`; do echo "$i"; done
for i in `ls *`;do head "$i" | cut -f 3 ; done
  • sed
cat 1.gtf | head |sed 's/1802/12/'
  • awk
cat 1-6.gtf |head | awk '{if($4>1000 && $5>2900)print}' | cut –f 3 | sort | uniq –c
  • history
Ctrl - r : search previous command
!V (im) : last command (be cautious !)
  • find and locate
find . -name "*.gtf"  # find . -name "pattern"
locate
  • xargs

grep for pattern in all files on the system:

find / | xargs grep pattern > out &

Move files in olddir to newdir, showing each command:

ls olddir | xargs -i  mv olddir/{} newdir/{}  
find . -type f -name "*.txt" | xargs -i  mv {} newdir/{}

2) Tips

(1) bashrc and .bash_profile

Example:

# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi


# User specific environment and startup programs
if [ -f $HOME/shortcuts ]; then
        source $HOME/shortcuts
fi
PATH=$HOME/bin:$PATH
export PATH


# User specific aliases and functions
alias qstat="qstat -u '*'"
#alias screen="/usr/bin/screen -D -R"
#alias rm="$HOME/bin/del.sh"
#alias undel="$HOME/bin/del.sh -u"
#alias ls="ls --color"
alias ld="ls -d"
alias c="clear"
alias l="ls -alh"
alias lf="ls -F|grep /"
alias lt="ls -tlr"
alias mv="mv -i"
alias cp="cp -pi"
alias diff="diff -b"


#PERL5LIB=$MYHOME/perllib:$MYHOME/perllib/lib64/perl5/site_perl/5.8.5:$MYHOME/perlib/lib/perl5/site_perl/5.8.5
#export PERL5LIB
#export R_LIBS_USER=~/R:/data/apps/R_library

more examples

(2) nohup, screen and qsub

  • run something at background

./run.bat >& run.log &

fg

  • nohup:

nohup nice -19 run.bat >& run.log&

  • tmux:
    • start a new session: tmux or tumx new-session -s session-name
    • detach: ctrl-a, d
    • re-attach: tmux attach-session -d -t session-name #detach it first
  • screen: # a popular alternative of tmux
    • start a new session: screen
    • detach: ctrl-a, d
    • re-attach: screen -R -D # detach it first
  • qsub:
    • qlogin not allowed in some servers
    • Check qstat -u '\*'
    • example script

(3) secure your files

  • make your files Read-only
    • permission for a executable bash script is usually 755
    • using chmod -R a-w for raw data and input files
  • 777, rwxrwxrwx is forbidden (using chmod -R o-w)
  • Change user group and permission
usermod –G admin  john  #  add sudo

usermod –G lulab  liyang  #  add to lulab group
chgrp –R  lulab  yourdir/
chmod –R  g+w   yourdir/

chmod  a+x  yourdir/
chmod  o+x  yourdir/
chmod  o-w  yourdir/
  • Blocking the root:

vim /etc/ssh/sshd_config
#PermitRootLogin yes —>no

(4) Setup ssh key

  • ssh-keygen (authorized_keys id_rsa.pub 权限设置为 600)
ssh-keygen -t rsa
copy authorized key in ~/.ssh/id_rsa.pub to remote_machine:~/.ssh/athorized_keys

This will be very useful later when we work on remote machine as a local one, especially for jobs like backup and script editing.

(5) System

  • kernel version
uname -r 
uname -a
  • mount dir/ to local machine using NFS or sshfs (then you can Edit text and view figures remotely)

I recommend using "transmit" app in your mac instead of mounting NFS or sshfs.

$ yum install fuse-sshfs
$ usermod -a -G fuse john
sshfs john@nye:/Users/john /mnt/nyefs fusermount -u mountpoint
automounting: in /etc/fstab
sshfs#john@nye: /mnt/nyefs fuse uid=500,gid=100,allow_other 0 0
mount /mnt/nyefs; umnout /mnt/nyefs
  • forward the log file to an email address
vim /home/.forward or /root/.forward
  zhi_lu@tsinghua.edu
  • Kill batch job
top
ps -edalf | grep username 
kill -9 PID

3) More Readings and Practices

  • Teaching Video:Week I. 2. Linux
  • for Beginners
    • 阅读和练习《鸟哥的Linux私房菜-基础学习篇》如下章节:
    • 《“笨办法”学python》附录“命令行快速入门”

第5章
5.3.1 man page
第6章
6.1用户与用户组
6.2 LINUX文件权限概念
6.3 LINUX目录配置
第7章Linux文件与目录管理
7.1目录与路径
7.2文件与目录管理
7.3文件内容查阅
7.5命令与文件的查询
7.6权限与命令间的关系
第8章
8.2文件系统的简单操作
第9章
9.1压缩文件的用途与技术
9.2 Linux系统常见的压缩命令
9.3打包命令:tar
第10章vim程序编辑器 (或者其他编辑器文档)
第11章 认识与学习bash
第25章 LINUX备份策略
25.2.2完整备份的差异备份
25.3鸟哥的备份策略
25.4灾难恢复的考虑
25.5重点回顾

第11章 认识与学习bash
第12章 正则表达式与文件格式化处理
第13章 学习shell script

  • for Advanced Readers

《Bioinformatics Data Skills》

3) Remedial Unix Shell

7) Unix Data Tools

4) Homework

  1. 解释gtf/gff文件中第4、5列($4,$5)代表什么,exon长度应该是$5-$4+1还是$5-$4

  2. 从gtf/gff文件中寻找3个最长的exon:

    grep exon *.gtf | awk '{print $5-$4+1}' | sort -n | tail -3这个方法有什么bug?

    有新的方法加分,但必须注释清楚每个语句和参数的意义和结果。

  3. 从gtf/gff文件中寻找并计算每一个transcript的长度,注意不能重复计算,不能包含intron。

5) Video

5a) Linux

@youtube

@Bilibili

You can’t perform that action at this time.