Cluster client should retry gracefully

[mshuler]

Cluster client should retry gracefully for a reasonable amount of time when frontend server is stopped/restarted. I restarted our frontend server, with perhaps a 1 minute time gap between stop and start, and every client failed and needed to be manually restarted.

(env)ryan@bdplab0:~/git/cstar_perf$ supervisorctl status
crashmail                        RUNNING   pid 2085, uptime 39 days, 20:42:36
cstar_perf_client                FATAL     Exited too quickly (process log may have details)
(env)ryan@bdplab0:~/git/cstar_perf$ supervisorctl tail cstar_perf_client
cent call last):
  File "/home/ryan/git/cstar_perf/env/bin/cstar_perf_client", line 9, in <module>
    load_entry_point('cstar-perf.frontend==1.0', 'console_scripts', 'cstar_perf_client')()
  File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 821, in main
    job_runner.run()
  File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 203, in run
    self.__ws_client.connect()
  File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 70, in connect
    ws.connect(self.ws_endpoint)
  File "/home/ryan/git/cstar_perf/env/local/lib/python2.7/site-packages/websocket/_core.py", line 441, in connect
    raise err
socket.error: [Errno 111] Connection refused
DEBUG:cstar_perf.client:Connecting to ws://cstar.datastax.com/api/cluster_comms ...
Traceback (most recent call last):
  File "/home/ryan/git/cstar_perf/env/bin/cstar_perf_client", line 9, in <module>
    load_entry_point('cstar-perf.frontend==1.0', 'console_scripts', 'cstar_perf_client')()
  File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 821, in main
    job_runner.run()
  File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 203, in run
    self.__ws_client.connect()
  File "/home/ryan/git/cstar_perf/frontend/cstar_perf/frontend/client/client.py", line 70, in connect
    ws.connect(self.ws_endpoint)
  File "/home/ryan/git/cstar_perf/env/local/lib/python2.7/site-packages/websocket/_core.py", line 441, in connect
    raise err
socket.error: [Errno 111] Connection refused