-
Notifications
You must be signed in to change notification settings - Fork 0
Home
jdimatteo edited this page Nov 8, 2013
·
61 revisions
-
dfbradnerlab@gmail.com
- email address for sending email from TOD (e.g. for Jenkins)
- ask jdimatteo@gmail.com if you would like the password
- email list: bradnerlab-systemadmin@googlegroups.com
- email list for receiving email (e.g. if a backup fails, it will email this group, which will result in emails being sent to one or more individuals responsible for system admin)
- follow the instructions here to request being added to this list: https://groups.google.com/forum/?hl=en#!forum/bradnerlab-systemadminemail jdimatteo@gmail.com if you would like to be added to the group
-
http://tod.dfci.harvard.edu/jenkins
- install documented at misc-2
- tip: use for remote system admin, e.g.
- ssh -L 9001:tod.dfci.harvard.edu:80 harvard
- you can now access jenkins in a local web browser at http://localhost:9001/jenkins/
- only the bradneradmin user currently has permission to create/configure jobs
- the jenkins user has sudo permission to run some backup scripts in /root/backup-scripts/
- the jenkins user should still be safe to use for other jobs, and it would be fine to allow any authenticated user access to configure/create jenkins jobs
- configuration is carefully setup to NOT allow the jenkins user to run arbitrary backup commands (which would allow a user to backup files they don't have access to to arbitrary locations, or restore files they wouldn't normally have access to)
- the jenkins user should still be safe to use for other jobs, and it would be fine to allow any authenticated user access to configure/create jenkins jobs
- dfbradnerlab@gmail.com email used for sending messages
- summary: roughly every 48 hours rdiff-backup incremental backups to /crusader
- starts 3 AM every Tuesday, Thursday, and Saturday, run by Jenkins: http://tod.dfci.harvard.edu/jenkins/job/Backup_TOD/ (only viewable by the bradneradmin Jenkins user)
- error emails sent to bradnerlab-systemadmin@googlegroups.com
| TOD Path | Backup Path |
|---|---|
| /ark | /crusader/backup/rdiff-backup/ark/ |
| /mnt/d0-0/share/bradnerlab/ | /crusader/backup/rdiff-backup/mnt/d0-0/share/bradnerlab/ |
| /ifs/labs/bradner/ | /crusader/backup/rdiff-backup/ifs/labs/bradner/ |
| / --exclude /proc --exclude /sys --exclude /tmp --exclude /media --exclude /ark --exclude /raider --exclude /mnt --exclude /ifs --exclude /crusader | /crusader/backup/rdiff-backup/root_directory/ |
- above file paths backed up with rdiff-backup
- mysql backup with automysqlbackup
- backups up all mysql databases (as listed by "SHOW DATABASES") to /crusader/backup/automysqlbackup/
- background on automysqlbackup: http://sourceforge.net/projects/automysqlbackup/
- installation notes (note that I disabled the cron job, and later removed the performance_schema table to resolve an error as described here)
root@tod:~# apt-get install automysqlbackup
root@tod:~# cp /etc/default/automysqlbackup gunk/
root@tod:~# vim /etc/default/automysqlbackup
root@tod:~# diff gunk/automysqlbackup /etc/default/automysqlbackup
31c31
< BACKUPDIR="/var/lib/automysqlbackup"
---
> BACKUPDIR="/crusader/backup/automysqlbackup"
39c39
< MAILCONTENT="quiet"
---
> MAILCONTENT="stdout"
root@tod:~# mv /etc/cron.daily/automysqlbackup gunk/- the actual commands to do the backups are in scripts in /root/backup-scripts, which the Jenkins user has sudo access to
- note that Charles suggested that /crusader has good physical security, so this should be secure enough without encryption
- Common Errors
- "UpdateError" - this is a recoverable error, effecting just a single file that failed to be backed up because it was being modified during the backup. See http://rdiff-backup.stanford.edu/error_policy.html for more info. Usually no action needs to be taken, since all other files were backed up correctly and this single file will probably be backed up successfully during the next backup.
- TODO: schedule restore drills to verify integrity of backups
- some apt-get installed software on TOD (this isn't intended to be a complete list)
- automysqlbackup
- realpath
- iotop
- some python packages installed with easy_install:
- redis
- sphinx
- bokeh
- nice bash checker:
- TOD hardware
- 32 physical CPU cores running at 1.4 GHz (64 hyperthreading cores)
- AMD Opteron(TM) Processor 6272
- 4 CPUs, each with 8 cores, each with two threads
- 32 physical CPU cores running at 1.4 GHz (64 hyperthreading cores)
jdm@tod:~$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 64
On-line CPU(s) list: 0-63
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 4
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 21
Model: 1
Stepping: 2
CPU MHz: 1400.000
BogoMIPS: 4200.16
Virtualization: AMD-V
L1d cache: 16K
L1i cache: 64K
L2 cache: 2048K
L3 cache: 6144K
NUMA node0 CPU(s): 0-15
dm@tod:~$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 21
model : 1
model name : AMD Opteron(TM) Processor 6272
stepping : 2
microcode : 0x600063d
cpu MHz : 1400.000
cache size : 2048 KB
physical id : 0
siblings : 16
core id : 0
cpu cores : 8
apicid : 32
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bogomips : 4199.82
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb
...* about 128 GB of RAM:
jdm@tod:~$ free -g
total used free shared buffers cached
Mem: 125 123 2 0 0 121
-/+ buffers/cache: 2 123
Swap: 127 0 127
jdm@tod:~$