Every 10 seconds it will check the version returned by http://www.khanacademy.org/api/internal/dev/version and send a notification to HipChat if it has changed since the last check.
This runs as a process on our internal webserver, toby.
Since GAE will serve this URL from different instances for about an hour after a new default version is designated, we have added some buffering to alleviate flip-flopping. This behavior is documented at http://stackoverflow.com/questions/15416938/app-engine-version-served-by-default-appears-to-be-inconsistent-and-thrash-for
This means we cannot detect when an actual rollback to a previous version has occurred within a short window. But after 2 hours, we reset our history and can detect an inadvertant flip.
This uses alertlib (a sub-repo) to talk to hipchat. alertlib requires being able to import a file called secrets.py with the contents: hipchat_alertlib_token = ""
This service is controlled via upstart, so after modifying, you can restart like so: % sudo stop gae-default-version-notifier % sudo start gae-default-version-notifier