GET returned 403 rate limit exceeded

[aherrmann]

Describe the bug
On GitHub actions CI we frequently see errors of the form:

       ERROR: Traceback (most recent call last):
       	File "/private/var/tmp/_bazel_runner/d1e3bdaa0c37ce8d00ecff62b05849c3/external/rules_haskell/haskell/cabal.bzl", line 1690
       		_pin_packages(repository_ctx, <1 more arguments>)
       	File "/private/var/tmp/_bazel_runner/d1e3bdaa0c37ce8d00ecff62b05849c3/external/rules_haskell/haskell/cabal.bzl", line 1211, in _pin_packages
       		repository_ctx.download(<3 more arguments>)
       java.io.IOException: Error downloading [https://api.github.com/repos/commercialhaskell/all-cabal-hashes/git/ref/heads/hackage] to /private/var/tmp/_bazel_runner/d1e3bdaa0c37ce8d00ecff62b05849c3/external/stackage-pinning-test-unpinned/all-cabal-hashes-hackage.json: GET returned 403 rate limit exceeded
       INFO: Elapsed time: 16.647s
       INFO: 0 processes.
       FAILED: Build did NOT complete successfully (0 packages loaded)
       ERROR: Build failed. Not running target
       FAILED: Build did NOT complete successfully (0 packages loaded)
       INFO: Streaming build results to: https://app.buildbuddy.io/invocation/91044d8d-3ec0-4572-8456-7daab9d5fc3d
       FAILED: Build did NOT complete successfully (0 packages loaded)

Specifically

java.io.IOException: Error downloading
  [https://api.github.com/repos/commercialhaskell/all-cabal-hashes/git/ref/heads/hackage] to [...]:
  GET returned 403 rate limit exceeded

These are typically fixed by rerunning the tests. Unfortunately, GitHub actions does not allow rerunning individual checks only the full pipeline, so this may take a few reruns.

To Reproduce
Run GitHub actions CI on rules_haskell until the issue occurs.

Expected behavior
The pipeline should pass without flaky "403 rate limit" errors.

Environment

• OS name + version: Observed on Linux and macOS GH actions pipelines in both nixpkgs and bindist mode.
  • Mac OS X 10.15.7 19H512
  • Ubuntu 20.04.2 LTS
• Bazel version: 3.3.1 and 3.7.2
• Version of the rules: Since Move macOS CI to GitHub Actions #1460

Additional context
This issue seems specific to GH actions, see #1460 (comment). I have not seen this error on CircleCI or locally.
The attempt to avoid this by reducing downloads via a repository cache does not seem to work reliably.

The failing fetch typically occurs during the pinning test (nixpkgs, bindist).

[aherrmann]

The issue may be caused by unauthenticated Github API requests. According to the documentation unauthenticated requests are rate limited to 60 per hour. Authenticated requests on the other hand are limited to 5k-15k requests per hour depending on the type of account.

According to the Github actions documentation Github generates a fresh access token for each workflow and makes it available through a secret. I'm not sure whether this token incurs it's own per workflow rate limit or applies to the accounts overall rate limit. Either way it should allow more than the 60 requests per hour and should hopefully fix our issue.