[MC-622] Handle TooManyRequestsError when sending emails

[Masergio]

There are rate limits on how frequently it can be called the v3 Mail Send endpoint. Currently, it may take up to 10,000 requests per second to send endpoint.
When it is reached a rate limit you will no longer be able to make request against that endpoint for the duration of that refresh period. Once this threshold has been reached, Sendgrid will return a status code 429 response (TOO MANY REQUESTS).

[shyaniv7]

The RateLimit error returns the time it requires to wait before being able to send emails again. To be on the safe side we should at the minimum sleep for that duration + buffer

What do you all think? or does everything think that 1 is sufficient because the limit is per second anyway?

In addition, rate limit lets us know how many were sent and how many can be sent. Should we use that data to make sure we don't "over" send?

[Masergio]

@shyaniv7

1. It seems that not all endpoints return rate limit headers to lets us know how many were sent and how many can be sent
   This guy also did not receive these headers for send mail endpoint

2. In case if rate limit headers are shown we can process next headers:

X-RateLimit-Limit: 500  # limit for some endpoint
X-RateLimit-Remaining: 499  # remaining requests for the endpoint
X-RateLimit-Reset: 1392815263  # looks like timestamp 

It is unobvious how we can use this information with the send-mail endpoint. Remaining rate limit for send-mail endpoint updates every second, for me it is overhead to try to handle it in more complex way

Also here is discussion of implementation X-RateLimit headers and proposition to show Reset in delta-seconds instead of timestamp It makes sense in my opinion

3. We also have the ability to monitor the overflow of the rate-limit using logs in datadog and if this is not enough, we will improve the TooManyRequestsError handling.