failing integration tests

[chris48s]

📋 Description

sigh

5 days ago we merged #8423
First dependabot barrage comes along, we've got failing builds with "API rate limit exceeded", "Error: Token pool is exhausted", etc
The token that GitHub allocates for workflow builds has a rate limit of 1,000 requests per hour. That's less than a PAT, but still quite a lot. I'm surprised we are managing to hit it, but we are.
I pushed a build to a branch that calls https://api.github.com/rate_limit just to check that the token definitely has a rate limit of 1,000 and we're definitely maxing it.

We are

{
  "resources": {
    "core": {
      "limit": 1000,
      "used": 1000,
      "remaining": 0,
      "reset": 1665865803
    },
    "search": {
      "limit": 30,
      "used": 0,
      "remaining": 30,
      "reset": 1665864671
    },
    "graphql": {
      "limit": 1000,
      "used": 0,
      "remaining": 1000,
      "reset": 1665868211
    },
    "integration_manifest": {
      "limit": 5000,
      "used": 0,
      "remaining": 5000,
      "reset": 1665868211
    },
    "source_import": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1665864671
    },
    "code_scanning_upload": {
      "limit": 1000,
      "used": 0,
      "remaining": 1000,
      "reset": 1665868211
    },
    "actions_runner_registration": {
      "limit": 10000,
      "used": 0,
      "remaining": 10000,
      "reset": 1665868211
    },
    "scim": {
      "limit": 15000,
      "used": 0,
      "remaining": 15000,
      "reset": 1665868211
    },
    "dependency_snapshots": {
      "limit": 100,
      "used": 0,
      "remaining": 100,
      "reset": 1665864671
    }
  },
  "rate": {
    "limit": 1000,
    "used": 1000,
    "remaining": 0,
    "reset": 1665865803
  }
}

sooo.. I need to work out what we're going to do about that.

One obvious problem is that the integration build is running on both push and pull-request. Dependabot pushes to this repo, rather than a fork which means the builds are doubling up so we can probably half the usage by fixing that. Looking at the number of failures we have, that will not be a complete solution.

This will also get worse once we've got danger and the service tests sharing the same token.

[chris48s]

Obviously when we get to merging stuff, this situation is also exacerbated by the fact that we run the tests lots of times because repo ranger updates each PR in the queue to make sure it is up to date with the base branch..and then of course, as well as running the integration tests on 2 node versions (16 and 17) on the PR (pointlessly twice at the moment), we run them again after we merge it to master as well.

[chris48s]

note: also look at close-bot / pull_request_target events and actions/dependency-review-action