AustinAppel.tex

\chapterwithauthor{Austin Appel}{Backups to Maintain Sanity}

\authorbio{Austin ``scorche'' Appel is an information security professional who spends
his time breaking into things previously thought secure (with permission, of 
course). He is often seen around security/hacker conferences teaching people how
to pick locks. In the open source world, he does a host of things for the Rockbox
project and previously volunteered for the One Laptop Per Child project.}

\noindent{}Backups are good. Backups are great. A competent admin always keeps current
backups.  This much can be gathered from any manual on server administration.
The problem is that backups are only really used when absolutely necessary.
If something drastic happens to the server or its data and one is forced to fall
back on something, the backups will come to the rescue in the moment of most
dire need. However, this should never happen, right? At any other time, what does
having backups do for you and your server environment?

Before going further, it is important to note that the advice espoused is for
the smaller open source project server administrators out there -- the silent
majority. If you maintain services that would cause a large amount of
frustration, and even perhaps harm if they experienced any downtime, please take
this with a very small grain of salt.

For the rest of us who work with countless smaller projects with limited
resources, we rarely have a separate test and production server. In fact, with
all of the many services that an open source project typically needs to maintain
(version control, web services, mailing lists, forums, build bots, databases,
bug/feature trackers, etc.), separate testing environments are often the stuff
we can only dream about. Unfortunately, the typical approach to system
administration is to tread lightly and only upgrade the systems when absolutely
necessary, to avoid risking dependency issues, broken code, or any of the other
million things that could go wrong. The reason you are nervous is not because
you may be inexperienced. It is important to know that you are not alone in
this. Whether or not we admit it to others, many of us have been (and likely
still are) in this position. The sad fact is that this inaction -- stemming from
the fear of breaking a ``working'' system -- often leads to running services which 
are often several versions behind the curve, and come with a host of potentially
serious security vulnerabilities. Rest assured that this is not the only way to play
the game though.

People tend to play a game differently when they have infinite lives as compared
to needing to start over from the start as soon as one mistake is made. Why
should server administration be any different? Approaching the concept of
backups with an offensive mindset can change your whole view of administrating
systems. Instead of living in fear from a complete dist-upgrade (or equivalent
command for yum, pacman, etc.), when armed with backups, one is free to update
the packages on a server secure in the knowledge that the changes can always be
rolled back if things turn sour. The key to getting over this is all about a
state-of-mind. There is no reason to fear as long as you have your safety net of
backed-up files beneath you as you jump. After all, system administration is
constantly a learning experience.

Of course, if you do not validate your backups, relying on backups in this way
becomes a very dangerous game.  Fortunately, experienced system administrators
know that the commandment ``keep current backups'' is always followed by ``validate
your backups.'' Again, this is another mantra that people like to recite.  What
does not fit as elegantly into a catchy mantra is how quickly and easily
validating backups can be accomplished. The best way to tell that a backup works
is, of course, to restore it (preferably on an identical system not currently
active). But again, in the absence of such luxuries, a bit more creativity is
required. This is where (at least for files) checksums can help you determine
the integrity of your backed-up files. In rsync, for example, the default method
it uses to determine which files have been modified is to check the time of last
modification and size of the file.  However, by using the “-c” option, rsync
will use a 128-bit MD4 checksum to determine whether files have changed or not.
While this may not always be the best idea to do every time in all situations
due to likely taking much longer than a normal rsync and increased io usage,
this ensures that the files are intact.

The role of system administrator can be a stressful one at times.  However,
there is no need to make it more so than it needs to be.  With the proper frame
of mind, some ostensibly single-purpose defense-seeming tasks can be used as
valuable tools to allow you to nimbly move forward with your sanity intact with
the speed appreciated by all open source projects.