Throttling: Rate exceeded when deploying in CI/CD

[fean]

Describe the bug
We have a simple Next.js app. It has two pages and no API lambda's at this moment. Deploying it locally works like a charm.

When we moved it to our Gitlab CI pipeline. It started throwing the following error consistently:

Throttling: Rate exceeded
     at Request.extractError (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/protocol/query.js:50:29)
     at Request.callListeners (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
     at Request.emit (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
     at Request.emit (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/request.js:683:14)
     at Request.transition (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/request.js:22:10)
     at AcceptorStateMachine.runTo (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/state_machine.js:14:12)
     at /root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/state_machine.js:26:10
     at Request.<anonymous> (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/request.js:38:9)
     at Request.<anonymous> (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/request.js:685:12)
     at Request.callListeners (/root/.serverless/components/registry/npm/serverless-next.js@1.13.0/node_modules/aws-sdk/lib/sequential_executor.js:116:18) 

{
   code: 'Throttling',
   time: 2020-06-12T10:21:05.448Z,
   requestId: 'dfdacc4e-9a19-4b7a-90cb-2930d565e773',
   statusCode: 400,
   retryable: true
 }

This is a masked representation of our serverless.yml file:

appStaging:
  component: serverless-next.js@1.13.0
  inputs:
    domain: [sub, domain.ext]
    name:
      defaultLambda: lambda-name-dev
      apiLambda: api-lambda-name-dev
    bucketName: bucket-name-dev
    memory:
      defaultLambda: 512
      apiLambda: 1024

The CI runner uses the node:12-alpine docker image to run the deployment.

To Reproduce
Steps to reproduce the behavior:

1. Create a simple app
2. Do a deployment in Gitlab CI through npx serverless

Expected behavior
We expect the pipeline to deploy our application as it was locally.

Desktop (please complete the following information):

• OS: Alpine Linux
• Version: 3.11

[aidan-ayala]

+1

[danielcondemarin]

Hi 👋
Could you do something along these lines to find which API call is getting throttled?
https://aws.amazon.com/premiumsupport/knowledge-center/cloudtrail-rate-exceeded/

[aidan-ayala]

Here's my serverless.yml. Happens on both CI and locally after running serverless --stage=staging

myAppName: component: serverless-next.js

[vkeskinen]

Could it be the ongoing IAM issue (at least for us it is affecting all our serverless deployments with error rate exceeded):

We have identified the root cause of the increased error rates and latencies on the AWS IAM CreateRole and CreateServiceLinkedRole APIs and are working towards resolution.

https://status.aws.amazon.com/

[danielcondemarin]

Thanks @vkeskinen that seems like the culprit. I’ll close this and feel free to reopen if its not resolved after Iam status is green.

[juancjara]

Hi @danielcondemarin , thanks for the awesome job on this library, I'm also facing the issue, so as I read from the comments it's related to aws, so there is no extra config for fixing it, right?

[MGough]

I also had this issue on Github Actions. The previous logged line was: DEBUG ─ Configuring DNS for distribution. There weren't any reported isssues at the time, and there still aren't any showing on the AWS Service Status Dashboard ~15 hours later.

   error:
  Throttling: Rate exceeded
    at Request.extractError (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/protocol/rest_xml.js:53:29)
    at Request.callListeners (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/request.js:688:14)
    at Request.transition (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/request.js:690:12)
    at Request.callListeners (/home/runner/.serverless/components/registry/npm/@sls-next/serverless-component@1.17.0/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'Throttling',
  time: 2020-10-22T17:10:05.448Z,
  requestId: '2b4cf2de-a09e-47bd-baec-2e3dc18fad8c',
  statusCode: 400,
  retryable: true
}
25h
  83s › bitfountHub › Throttling: Rate exceeded

Strangely I don't see any error codes when following the guide above or when searching through CloudTrail.

[danielcondemarin]

Hi @danielcondemarin , thanks for the awesome job on this library, I'm also facing the issue, so as I read from the comments it's related to aws, so there is no extra config for fixing it, right?

That's right, there is not much serverless-next.js can do here. This is AWS throttling your API requests. Generally this is to do with a lot of API calls being made within the same account in bursts. Each different service has different quotas so you'll need to look which one is exceeding the rate limit and ask maybe for an increase. If you search the AWS docs there is plenty of info on this.

[patricktyndall]

Did anyone manage to fix this?

[Gonzasestopal]

Happens intermittently using github actions for me.

Tried debugging cloudtrail but couldnt find anything useful.

Might be helpful knowing which services are most likely to exceed these default quotas.

version: @sls-next/serverless-component@1.19.0-alpha.37

[patricktyndall]

Some info:

• I ran my deploy script (outside of Jenkins) from a different EC2 instance (same subnet, SGs, etc) and it works. (This probably eliminates IAM as a culprit).
• This supports IP-based limits (simplest explanation), but here's an argument against that: If that's the case (knowing that my failing instance gets 0 activity outside of these deploys), it seems not to be a burst or time-based limit, so much as a permanent limit for the specific IP address, which would be weird. (Also if IP-based, you'd expect to see this happen more often from shared infra like GH Actions.)