AppController crashes in response to failed ZooKeeper operations

[nlake44]

If a ZooKeeper write fails, we throw a FailedZooKeeperOperationException. A stack trace showing this is:

/root/appscale/AppController/lib/zkinterface.rb:638:in `set': Failed to set path /appcontroller/state with data 
{:rc=>-7, :stat=>#<ZookeeperStat::Stat:0x7fc864a1e0a8 @exists=false>, :req_id=>11026} (FailedZooKeeperOperationException)
        from /root/appscale/AppController/lib/zkinterface.rb:178:in `write_appcontroller_state'
        from /root/appscale/AppController/djinn.rb:1461:in `backup_appcontroller_state'
        from /root/appscale/AppController/djinn.rb:795:in `job_start'
        from /root/appscale/AppController/djinnServer.rb:120

This crashes the AppController, and when it restarts, it removes all the nginx config files on this machine, which makes users unable to access their applications. The AppController fails to restart correctly, so users are never able to access their applications.

[shatterednirvana]

Updated description above. As the description above indicates, this issue is really multiple bugs:

1. Failed ZooKeeper operations crash the AppController (this issue).
2. The AppController erases nginx config files when it starts up.
3. The AppController doesn't revive properly. This is a known issue (AppController does not properly revive upon failure #33)

Opened issues for 2 (#67), and continuing to investigate this issue.

[shatterednirvana]

The branch proposed above should fix this problem. Fixed it by manually stating a longer timeout between our client and Zookeeper (thus avoiding SessionExpired exceptions), and by retrying in cases where we couldn't successfully write to Zookeeper (instead of raising FailedZooKeeperOperationException). We now try five times to do a write and then simply skip on the write.

In the future, we should revisit when simply skipping on the write is ok, and when it isn't.

[shatterednirvana]

QA'ed and merged in!