"CustomError: Access to requested resource is forbidden" after a certain number of calls

[cs-pellsoftware]

We're building an app that loops a bunch of different ASINS and makes different API calls on each one. Everything works fine if it's just running up to 300 ASINS. However, if we try to run 800+ it starts giving this error:

CustomError: Access to requested resource is forbidden

Do I need to be doing rate limiting of some sort? Or is there another possible issue here?

[amz-tools]

Hi @cs-pellsoftware,

hard to say, but I would guess it could have something to do with the rate limit. We have experienced it in our development team that an excessive calling of the same operation in a too short amount of time without respecting the restore rate returns the expected QuotaExceeded error, but after some time it triggers the "access forbidden" error you also experienced. It seems as if Amazon starts to fully "block" these requests instead of returning the nicer "QuotaExceeded" error.

The client is taking care of rate limits to some extent, at least if you set auto_request_throttled to true (if you don't set it explicitly its true as well, since thats the default). If its set and an operation returns a "QuotaExceeded" error, we will wait for the restore rate to pass (coming either from x-amzn-ratelimit-limit header if given or as a fallback from the default restore rate defined in the SP docs) and then retry the request.

However what the client doesn't have (yet) is something like a "smart queue". I.e. lets say you start 20 requests of the same operation in parallel whereas the operation has a burst of 10 requests and a restore rate of 2 seconds. What will happen then is Amazon is returning a successful response for 10 of them and a QuotaExceeded error for the remaining 10. These 10 requests will be retried after the restore rate of 2 seconds. This would cause 1 request to return a successful response and 9 of them being throttled once more. Next time 8 will be throttled again and so on.

So to sum up, I could imagine that this might have happened here for you, ultimately resulting in Amazon returning an "access forbidden" after some time (in your case after requesting about 300 asins).

So if that really is the issue here you would probably have to set auto_request_throttled to false and handle the rate limit logic on your end.

[amz-tools]

As of version 0.7.1 we have a new option to overwrite the default restore_rate for an operation by setting it inside the options of .callAPI(). This should bring some more flexibility to prevent such errors (compare issue #101 as well). So I think we can close this for the time being.