Deploying multiple runners fails with github rate limit after version 0.5.0

[thatGreekGuy96]

Hey everyone,
Noticed the following error appearing from version 0.5.0 onwards. We have a stage in our github actions pipeline that deploys multiple runners, as such:

  deploy_runners:
    name: Deploy Cloud Instances
    needs: [setup_config]

    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v2
      - uses: iterative/setup-cml@v1
        with:
          version:  ${{ env.cml_version }}

      - name: "Deploy runner on EC2"
        shell: bash
        env:
          repo_token: ${{ secrets.ACCESS_TOKEN_CML }}
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID_TESTING }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY_TESTING }}
          CASE_NAME: ${{ matrix.case_name }}
          N_RUNNERS: ${{ fromJson(needs.setup_config.outputs.json_string).n_runners }}

        run: |
          for (( i=1; i<=N_RUNNERS; i++ ))
          do
            echo "Deploying runner ${i}"
            cml-runner \
            --cloud aws \
            --cloud-region eu-west-2 \
            --cloud-type=m \
            --cloud-hdd-size 100 \
            --cloud-spot \
            --labels=cml-runner &
          done
          wait
          echo "Deployed ${N_RUNNERS} runners."

I noticed that the following error has started appearing after version 0.5.0 of cml was released:

2021-07-11T12:25:17.9903383Z RequestError [HttpError]: You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.
2021-07-11T12:25:17.9905919Z     at /usr/local/lib/node_modules/@dvcorg/cml/node_modules/@octokit/request/dist-node/index.js:86:21
2021-07-11T12:25:17.9907296Z     at processTicksAndRejections (internal/process/task_queues.js:95:5)
2021-07-11T12:25:17.9909090Z     at async Object.next (/usr/local/lib/node_modules/@dvcorg/cml/node_modules/@octokit/plugin-paginate-rest/dist-node/index.js:124:28)
2021-07-11T12:25:17.9910621Z     at async Github.getRunners (/usr/local/lib/node_modules/@dvcorg/cml/src/drivers/github.js:233:17)
2021-07-11T12:25:17.9911889Z     at async CML.getRunners (/usr/local/lib/node_modules/@dvcorg/cml/src/cml.js:221:12)
2021-07-11T12:25:17.9913066Z     at async CML.runnerByName (/usr/local/lib/node_modules/@dvcorg/cml/src/cml.js:226:21)
2021-07-11T12:25:17.9914381Z     at async run (/usr/local/lib/node_modules/@dvcorg/cml/bin/cml-runner.js:326:7) {
2021-07-11T12:25:17.9915289Z   status: 403,
2021-07-11T12:25:17.9916315Z   response: {
2021-07-11T12:25:17.9917781Z     url: 'https://api.github.com/repositories/263006174/actions/runners?page=8',
2021-07-11T12:25:17.9918908Z     status: 403,
2021-07-11T12:25:17.9919585Z     headers: {
2021-07-11T12:25:17.9920423Z       'access-control-allow-origin': '*',
2021-07-11T12:25:17.9923366Z       'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
2021-07-11T12:25:17.9926135Z       connection: 'close',
2021-07-11T12:25:17.9926949Z       'content-encoding': 'gzip',
2021-07-11T12:25:17.9937883Z       'content-security-policy': "default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events translator.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com render-temp.githubusercontent.com viewscreen.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com secured-user-images.githubusercontent.com/; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com",
2021-07-11T12:25:17.9947806Z       'content-type': 'application/json; charset=utf-8',
2021-07-11T12:25:17.9948751Z       date: 'Sun, 11 Jul 2021 12:25:17 GMT',
2021-07-11T12:25:17.9950048Z       'expect-ct': 'max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"',
2021-07-11T12:25:17.9951813Z       'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
2021-07-11T12:25:17.9953126Z       'retry-after': '60',
2021-07-11T12:25:17.9953893Z       server: 'GitHub.com',
2021-07-11T12:25:17.9955102Z       'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
2021-07-11T12:25:17.9956369Z       'transfer-encoding': 'chunked',
2021-07-11T12:25:17.9957547Z       vary: 'Accept-Encoding, Accept, X-Requested-With',
2021-07-11T12:25:17.9958535Z       'x-content-type-options': 'nosniff',
2021-07-11T12:25:17.9959415Z       'x-frame-options': 'deny',
2021-07-11T12:25:17.9960321Z       'x-github-media-type': 'github.v3; format=json',
2021-07-11T12:25:17.9961382Z       'x-github-request-id': '04FC:02AD:4BF0BFD:907BDD4:60EAE32D',
2021-07-11T12:25:17.9962208Z       'x-xss-protection': '0'
2021-07-11T12:25:17.9962722Z     },
2021-07-11T12:25:17.9963157Z     data: {
2021-07-11T12:25:17.9964543Z       documentation_url: 'https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#abuse-rate-limits',
2021-07-11T12:25:17.9966598Z       message: 'You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.'
2021-07-11T12:25:17.9967480Z     }
2021-07-11T12:25:17.9967921Z   },
2021-07-11T12:25:17.9968390Z   request: {
2021-07-11T12:25:17.9969032Z     method: 'GET',
2021-07-11T12:25:17.9970129Z     url: 'https://api.github.com/repositories/263006174/actions/runners?page=8',
2021-07-11T12:25:17.9970995Z     headers: {
2021-07-11T12:25:17.9971876Z       accept: 'application/vnd.github.v3+json',
2021-07-11T12:25:17.9973977Z       'user-agent': 'octokit-core.js/3.5.1 Node.js/14.17.1 (linux; x64)',
2021-07-11T12:25:17.9975171Z       authorization: 'token [REDACTED]'
2021-07-11T12:25:17.9977395Z     },
2021-07-11T12:25:17.9978105Z     request: { agent: [Agent], hook: [Function: bound bound register] }
2021-07-11T12:25:17.9978736Z   }
2021-07-11T12:25:17.9979202Z }

Any clues as to what would be going on? Reverting to 0.4.7 fixed the problem.

[DavidGOrtega]

Its GH limiting the api requests... It seems to be requesting lots of pages in your case. Originally we were not paginating since one page of online runners sufficed.

It should be an easy fix 🙏

[0x2b3bfa0]

Are we really performing that many requests? Authenticated calls have generous limits.

[DavidGOrtega]

It seems so! Or at least its what the error says.

You have triggered an abuse detection mechanism.
...
 status: 403,
2021-07-11T12:25:17.9916315Z   response: {
2021-07-11T12:25:17.9917781Z     url: 'https://api.github.com/repositories/263006174/actions/runners?page=8',
2021-07-11T12:25:17.9918908Z     status: 403,

[0x2b3bfa0]

Eight pages of runners? 🙀

[DavidGOrtega]

https://docs.github.com/en/rest/overview/resources-in-the-rest-api#abuse-rate-limits
https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-abuse-rate-limits

[DavidGOrtega]

this is very interesting.
I had to do 5000 request in a row to raise the error @thatGreekGuy96

const promises = [];
 for (let i = 0; i < 5000; i++) {
   promises.push(cml.runnerByName({ name }));
 }

 await Promise.all(promises)

sounds to me a lot! Are you triggering that much events @thatGreekGuy96 ? We might be loosing something

[thatGreekGuy96]

As far as I'm aware, I'm just calling 'cml-runner' 30 times, so nowhere near 5000. I don't know how many calls to GitHub's API happen within that command though.

[DavidGOrtega]

30 x 8 pages... that I know based on your logs

[0x2b3bfa0]

@thatGreekGuy96, given that all the calls are being made concurrently from the different cml-runner processes, I wouldn't be too surprised if GitHub starts to complain. Not just because of the quantity, but also proabably because of the density: each of these processes is issuing too many API calls at the same time.

Maybe you could send them in batches. It's going to be slower, but probably not that much.

for ((runner=0; runner<runners; runner++)); do ((runner%10)) || wait
  cml-runner ··· &
done
wait

[thatGreekGuy96]

I'll give that a try, it shouldn't have too big of an impact.

[DavidGOrtega]

Im finishing the PR adding throttling and being smarter when consuming it