Async events processing

[janisz]

HttpEventActor try to be smart and if specific subscriber is not responding then rate to this subscriber is limited.

When consul is not responding (e.g., there is no leader, hos is unreachable, timeout occurs) we pass this error to Marathon returning 500. In the end rate to marathon-consul is limited and no events are sent hence service is not de/registered.

E.g.:

echo '{
          "slaveId":"85e59460-a99e-4f16-b91f-145e0ea595bd-S0",
          "taskId":"web",
          "taskStatus":"TASK_LOST",
          "message":"Command terminated with signal Terminated",
          "appId":"/test/app",
          "host":"localhost",
          "ports":[
            31372
          ],
          "version":"2015-12-07T09:02:48.981Z",
          "eventType":"status_update_event",
          "timestamp":"2015-12-07T09:33:40.898Z"
}' |  \
  http POST http://localhost:4000/events \
  cache-control:no-cache \
  content-type:application/json \

HTTP/1.1 500 Internal Server Error
Content-Length: 115
Content-Type: text/plain; charset=utf-8
Date: Wed, 20 Apr 2016 12:20:01 GMT

Put http://127.0.0.1:8500/v1/agent/service/deregister/web: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

To solve this issue we should return 202 Accepted and process events asynchronous. As a first step we could change 500/400 responses to 200, but it will not prevent rate limit on consul timeout.
Changes are required in handler. Remember to check metrics and error handling.