Skip to content

UNIX

Matthew Hutchinson edited this page Nov 9, 2015 · 14 revisions

Th basics of the Unix Philosophy (extracted from here)

  • Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.
  • Rule of Modularity: Write simple parts connected by clean interfaces.
  • Rule of Clarity: Clarity is better than cleverness.
  • Rule of Composition: Design programs to be connected to other programs.*
  • Rule of Separation: Separate policy from mechanism; separate interfaces from engines.*
  • Rule of Simplicity: Design for simplicity; add complexity only where you must.
  • Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.*
  • Rule of Transparency: Design for visibility to make inspection and debugging easier.
  • Rule of Robustness: Robustness is the child of transparency and simplicity.
  • Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.
  • Rule of Least Surprise: In interface design, always do the least surprising thing.
  • Rule of Silence: When a program has nothing surprising to say, it should say nothing.
  • Rule of Repair: When you must fail, fail noisily and as soon as possible.
  • Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
  • Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
  • Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
  • Rule of Diversity: Distrust all claims for “one true way”.
  • Rule of Extensibility: Design for the future, because it will be here sooner than you think.

mail issues and exmin

Check for service sendmail restart, or /etc/init.d/exim restart (or sendmail). Check /var/syslog or /var/mail.err|warn|log for issues.

mailq # view the current queue
exim -v -M 8732t7823r9823h # force send a message using id from mail (can see exim issues)
mailq | grep frozen | awk '{print $3}' | xargs exim -v -M  # send all frozen messages

# force clear the queue (will delete unsent queued emails)
rm /var/spool/mqueue/*

tmux stream/share

tmux -S /tmp/team # host
tmux -S /tmp/team attach # client

Timezone

date
sudo  ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
date

Passenger

Memory status with passenger, (tweaking PassengerMaxPoolSize with available memory)

passenger-status (check Waiting on global queue)
passenger-memory-stats (check private memory on your workers)
free -m
vmstat 2 (check for si,so swap in/out)

Xargs

Good examples of xargs usage.

gem list --no-versions | xargs --interactive sudo gem uninstall -a -I

Capistrano - setting remote ENV variables

When using no standard path (e.g. different ruby) on remote server then adjust $HOME/.ssh/environment. You do not need to export the var, just insert a single line like this:

PATH=/opt/csw/bin:/usr/bin:/usr/sbin:/usr/ucb:/usr/ccs/bin:/blah/blah/blah

Edit your sshd_config or SSH won't even read that file. So, as root edit /etc/ssh/sshd_config and add the following line to the end of the file: PermitUserEnvironment yes Then restart ssh then cap shell and try echo $PATH

Using PV (pipe viewer) to track progress

For example, bzip extraction of an approx. 196Mb file piped to mysql insertion

apt-get install pv
# or 
brew install pv
# example
cat ./ticketsolve.db.production.20091102-040002.sql.bz2 | pv -s 196M | bzip2 -d | mysql -uroot ticketsolve_rightscale
# another example
tail -f log/development.log | grep Completed | pv -lr > /dev/null
# -l (count lines rather than bytes)
# -r (show rate info)

Regular tasks

  • check version; cat /etc/lsb-release
  • look at sources; sudo nano /etc/apt/sources.list
  • look at startup processes; ls -la /etc/rc?.d/S* (or install rcconf)

Clean out big files taking space

E.g. the top 10 largest files in any directory

cd /path/to/some/where
du -hsx * | sort -rh | head -10

Update System

  • update sources; sudo aptitude update
  • check for upgrade; sudo aptitude safe-upgrade
  • do upgrade; sudo aptitude full-upgrade

Update Entire OS e.g. Gutsy -> Hardy

  • sudo aptitude search update-manager-core (check its installed)
  • sudo aptitude update (update packages)
  • modify sources list if any 404's
cd /etc/apt
sudo cp sources.list ./sources.list.bak
sudo vi sources.list
# (:%s/ie.archive/old-releases/g) etc.
# exclude security at bottom

sudo aptitude update
sudo aptitude safe-upgrade

update all that are kept back with;
aptitude install <paste-kept-back>
sudo aptitude safe-upgrade (check nothing else)

sudo cp sources.list.bak sources.list (put back sources list)
sudo do-release-upgrade << upgrade to hardy
# (check configs as the appear in prompts)
# cross fingers and wait until reboot finished

Configure SMTP (Postfix) - test connection & sending mail

sudo dpkg-reconfigure postfix
cat /etc/postfix/main.cf
cat /etc/postfix/master.cf

# reload with
sudo /etc/init.d/postfix reload

# check local sending client
readlink -f $(which sendmail)

SMTP authentication with Postfix

Testing SMTP with telnet

$ telnet mail.somewhere.com 25
 Trying 192.168.75.194...
 Connected to smtp.somewhere.com.
 Escape character is '^]'.
 220 smtp.somewhere.net ESMTP Sendmail 8.9.3+Sun/8.9.1; Thu,

    12 Oct 2000 04:39:40 -0700 (PDT)
 helo aplawrence.com
 250 smtp.somewhere.com
 mail from: tony@aplawrence.com
 250 tony@aplawrence.com... Sender ok
 rcpt to: foobah@aplawrence.com
 550 foobah@aplawrence.com... Relaying denied
 rcpt to: foobah@somewhere.com
 250 foobah@somewhere.com... Recipient ok
 data
 354 Enter mail, end with . on a line by itself
 test
 look ma no headers!
 .
 250 HA00945 Message accepted for delivery
 quit
 221 smtp.somewhere.com closing connection

Monitor for Process then restart if not found

#!/bin/bash
# Server Process Monitor

# get in the right directory
WORKING_DIR=/u/apps/beyondendurance/current
cd $WORKING_DIR

# restart command to run
RESTART="/usr/local/bin/ruby script/mail_fetcher start"

# path to pgrep command with args
PGREP="/usr/bin/pgrep -f"

# daemon name
PROCESS="script/mail_fetcher"

# find pid
$PGREP ${PROCESS}

if [ $? -ne 0 ] # if not running
then
 # restart
 export RAILS_ENV=production
 set RAILS_ENV=production
 $RESTART
fi

SSH Keys

ssh-keygen -t rsa
ssh-keygen -t dsa
# or for an existing SSH Key, copy your public key to the server with
scp ~/.ssh/id_rsa.pub user@example.com:~/

Now on the server - append the public key to your authorized keys file and delete the file you uploaded:

mkdir .ssh
cat id_rsa.pub >> .ssh/authorized_keys
rm id_rsa.pub

# set permissions ok
chmod go-w ~
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

SHA1 checking with openssl sha1 path/to/file

Scraping Websites with wget

Or if grabbing an entire site from the web use these args to fake out the server;

wget --wait=20 --limit-rate=20K -r -p -U Mozilla --convert-links --html-extension -r http://grabthissite.com

wget --convert-links --html-extension -r http://localhost:3000/

--no-parent is a very handy option that guarantees wget will not download anything from the folders beneath the folder you want to acquire

Actually, to download a single page and all its requisites (even if they exist on separate websites), and make sure the lot displays properly locally, this author likes to use a few options in addition to ‘-p’: "

wget -E -H -k -K -p http://grabthissite.com

Emailing

cat report.txt | mail -s "subject here" matt@hiddenloop.com 
sendmail matt@hiddenloop.com < testmail.txt

# where testmail.txt is something like;
# to: matt@email.com
# subject: im the subject
# from: blingbling@from.com
# i am the body

Maintenance and useful one liners

perl -i -ne ' s/search_str/replace_str/; print ' some_file.txt # use perl to search & replace (with regex) file in place and resave
ruby -i -pe 'gsub "search_str", "replace_str"' my_file.txt     # or using ruby
sudo find . -type f -mtime +30 -exec rm {} +                   # delete files in dir, older than 30 days
find . -type d -name '.svn' -print0 | xargs -0 rm -rdf
find . -type f -exec grep -il "hello" {} \;
# or if you know the file type
find . -type f -name "*.ext" -exec grep -il "hello" {} \
and/ or if you just want your files
find . -type f -user myname -name "*.ext" -exec grep -il "hello" {} \
# and obviously you can still output this to a log file using the redirect >

Remote DB/Asset Backup

#!/bin/bash
if [ -e "dbdump.lock" ]
then
  echo "Lock file found (dbdump.lock). Please try again later (someone is dumping), or remove the file if you're sure nobody is doing it." >&2
else
  # backup database
  touch dbdump.lock
  db_backup_filename="bugle_$(date +%d)-$(date +%m)-$(date +%Y)_$(date +%H)-$(date +%M).sql.bz2"
  mysqldump bugle_production -ubugle -pwke3hfvuy1 | bzip2 -1 > "$db_backup_filename"
  scp "$db_backup_filename" buglebackup@herod.dreamhost.com:~/db

  # backup assets
  asset_backup_filename="bugle_$(date +%d)-$(date +%m)-$(date +%Y)_$(date +%H)-$(date +%M).tar.bz2"
  tar -cjf "$asset_backup_filename" /u/apps/bugle/shared/assets
  scp "$asset_backup_filename" buglebackup@herod.dreamhost.com:~/assets

  # cleanup
  rm "$asset_backup_filename"
  rm "$db_backup_filename"
  rm dbdump.lock
fi

Upstart

Include env vars in upstart exec (e.g. bluepill load). Place in /etc/init/pmfaqtory-store.conf.

# upstart for bluepill daemon and store app
# use with config/bluepill.pill

description "pmFAQtory store"

start on runlevel [2]
stop on runlevel [016]

exec su -c "$(tr "\n" " " < /etc/environment) bluepill load /var/www/pmfaqtory-store/current/config/bluepill.pill"

pre-stop exec bluepill stop

expect daemon

respawn

Useful links

Something went wrong with that request. Please try again.