How to perform an emergency rollback of the server

Have changes broken the server and you don't know how to immediately fix the problem? Here is how to perform a rollback to a previous commit:

# ssh into the server:
ssh sc-production@vps.suttacentral.net
cd suttacentral
# Rollback to a previous commit, the HEAD~1 means one commit previously, 
# HEAD~3 would go back three commits.
git checkout HEAD~1
# Restart the server
sudo supervisorctl restart sc-production

It is also possible to rollback the data repository in the same way, just perform the same steps in the suttacentral/data folder.

Once the problem has been fixed:

ssh sc-production@vps.suttacentral.net
cd suttacentral
git checkout master
exit

Then perform the invoke task to deploy changes to the server. If you aren't able to fix the problem yourself, inform the primary administrator that you've performed an emergency rollback so they know to perform the checkout step. Note that rolling back in this way has no affect on the git repository, it doesn't require reverting commits, it just makes the server use an older state of the files.