Deleting two API Gateway APIs in a row fails. TooManyRequestsException

[michael-dev2rights]

Let's say I have several APIs created by accident (in this case because I was testing things). I would like to delete them automatically.

When I do that I'll find that I get a timeout / TooManyRequestsException

In the attached ZIP file please find a simple script demo and the log of what happens when runing it.

I would normally expect this script to delete multiple APIs without problem and generally I'd expect boto3, as a high level interface, should hide the details of the rate limiting which the API does underneath.

Possibly this botocore issue Add possibility of modifying retry_config #891 would be helpful.

aws-TooManyRequestsException-broken-demo.zip

[michael-dev2rights]

Generally, this also seems to apply to lots of other API calls. I don't have simple examples to demo them yet though.

[JordonPhillips]

Looks like they use a slightly different http status code for that exception, which is why we didn't catch it automatically. I'll add that to our retry definition.

[JordonPhillips]

After a closer look I noticed that it is in fact already in our retry strategy - what version of botocore are you using? Looks like we added it January 7th. If your botocore version is up to date, that means that the request failed even with exponential backoff.

[michael-dev2rights]

@JordonPhillips I generated the bug on the latest version pulled from pip

$ pip freeze | grep boto
boto==2.43.0
boto3==1.4.1
botocore==1.4.67

Don't know if you have access to the internal cases; if you can see case ID [redacted] then you will see that the advice from AWS support is that the reason for the problem is not that there are different exception codes. It's that there are different rate limits on AkPI Gateway.

In particlular CreateRestApi, ImportRestApi, and DeleteRestApi are rate limited to 2 calls per minute.

As far as I can see from the exception (included in the log in the attached zip file) it has retried but it ran out of retries so that would match with what support is saying.

DeleteRestApi operation (reached max retries: 4): Too Many Requests

I'm guessing that this is because API Gateway has very low request per second limits. See the bottom of this document they linked to:

http://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

I have created a proposed fix for this in boto/botocore#1071

[JordonPhillips]

I'll have to discuss with the broader teams to see how we want to configure the retry strategy for these operations.

[michael-dev2rights]

@JordonPhillips thanks; waiting with baited breath.

From my side I think there are always going to be situations where e.g. you want to do more than one of any operation, so as a very minimum the longest retry should be longer than the longest you would have to wait after having done one operation. Ideally double that.

Even if you can't provide a fix soon, exposing a table of information about the rate limits would probably be helpfu.

[michael-dev2rights]

@JordonPhillips what was the result of your discussion. I've seen that failure to handle timeouts correctly is a problem for a large number of Amazon customers and is causing various problems with tools based on boto3 and in many discussion groups related to AWS configuration management. I'd be willing to put in a bit more work on this but not if you aren't interested in accepting it.

[shirishshukla]

Solution for me is add
time.sleep(50)

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than two years. We encourage you to check if this is still an issue in the latest release. Because it has been longer than two years since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.

[estahn]

Increasing max_attempts to trigger exponential backoff > 30s solved it.

Example:

client = boto3.client("apigatewayv2", config=Config(retries = dict(max_attempts = 10)))