Home

Gmail

• dfbradnerlab@gmail.com
  • email address for sending email from TOD (e.g. for Jenkins)
  • ask jdimatteo@gmail.com if you would like the password
• email list: bradnerlab-systemadmin@googlegroups.com
  • email list for receiving email (e.g. if a backup fails, it will email this group, which will result in emails being sent to one or more individuals responsible for system admin)
  • follow the instructions here to request being added to this list: https://groups.google.com/forum/?hl=en#!forum/bradnerlab-systemadminemail jdimatteo@gmail.com if you would like to be added to the group

Jenkins

• http://tod.dfci.harvard.edu/jenkins
  • install documented at misc-2
• tip: use for remote system admin, e.g.
  • ssh -L 9001:tod.dfci.harvard.edu:80 harvard
  • you can now access jenkins in a local web browser at http://localhost:9001/jenkins/
• only the bradneradmin user currently has permission to create/configure jobs
• the jenkins user has sudo permission to run some backup scripts in /root/backup-scripts/
  • the jenkins user should still be safe to use for other jobs, and it would be fine to allow any authenticated user access to configure/create jenkins jobs
    • configuration is carefully setup to NOT allow the jenkins user to run arbitrary backup commands (which would allow a user to backup files they don't have access to to arbitrary locations, or restore files they wouldn't normally have access to)
• dfbradnerlab@gmail.com email used for sending messages

Backups

• summary: roughly every 48 hours rdiff-backup incremental backups to /crusader
  • starts 3 AM every Tuesday, Thursday, and Saturday, run by Jenkins: http://tod.dfci.harvard.edu/jenkins/job/Backup_TOD/ (only viewable by the bradneradmin Jenkins user)
  • error emails sent to bradnerlab-systemadmin@googlegroups.com

TOD Path	Backup Path
/ark	/crusader/backup/rdiff-backup/ark/
/mnt/d0-0/share/bradnerlab/	/crusader/backup/rdiff-backup/mnt/d0-0/share/bradnerlab/
/ifs/labs/bradner/	/crusader/backup/rdiff-backup/ifs/labs/bradner/
/ --exclude /proc --exclude /sys --exclude /tmp --exclude /media --exclude /ark --exclude /raider --exclude /mnt --exclude /ifs --exclude /crusader	/crusader/backup/rdiff-backup/root_directory/

• above file paths backed up with rdiff-backup
  • background: http://www.backupcentral.com/wiki/index.php/Rdiff-backup
• mysql backup with automysqlbackup
  • backups up all mysql databases (as listed by "SHOW DATABASES") to /crusader/backup/automysqlbackup/
  • background on automysqlbackup: http://sourceforge.net/projects/automysqlbackup/
  • installation notes (note that I disabled the cron job, and later removed the performance_schema table to resolve an error as described here)

root@tod:~# apt-get install automysqlbackup
root@tod:~# cp /etc/default/automysqlbackup gunk/
root@tod:~# vim /etc/default/automysqlbackup
root@tod:~# diff gunk/automysqlbackup /etc/default/automysqlbackup 
31c31
< BACKUPDIR="/var/lib/automysqlbackup"
---
> BACKUPDIR="/crusader/backup/automysqlbackup"
39c39
< MAILCONTENT="quiet"
---
> MAILCONTENT="stdout"
root@tod:~# mv /etc/cron.daily/automysqlbackup gunk/

• the actual commands to do the backups are in scripts in /root/backup-scripts, which the Jenkins user has sudo access to
• note that Charles suggested that /crusader has good physical security, so this should be secure enough without encryption
• Common Errors
  • "UpdateError" - this is a recoverable error, effecting just a single file that failed to be backed up because it was being modified during the backup. See http://rdiff-backup.stanford.edu/error_policy.html for more info. Usually no action needs to be taken, since all other files were backed up correctly and this single file will probably be backed up successfully during the next backup.
• TODO: schedule restore drills to verify integrity of backups

Misc.

• some apt-get installed software (this isn't intended to be a complete list) ** automysqlbackup