503 error debugging

[robinelvin]

I have a CI build running in AWS CodeBuild which until a couple of days ago was working perfectly. Suddenly my builds failed with the following:

[Container] 2022/11/16 17:03:27 Running command yarn install --immutable
--
67 | Internal Error: Server answered with HTTP 503
68 | at ClientRequest.<anonymous> (/usr/local/lib/node_modules/corepack/dist/corepack.js:3933:31)
69 | at Object.onceWrapper (node:events:642:26)
70 | at ClientRequest.emit (node:events:527:28)
71 | at HTTPParser.parserOnIncomingClient [as onIncoming] (node:_http_client:631:27)
72 | at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
73 | at TLSSocket.socketOnData (node:_http_client:494:22)
74 | at TLSSocket.emit (node:events:527:28)
75 | at addChunk (node:internal/streams/readable:315:12)
76 | at readableAddChunk (node:internal/streams/readable:289:9)
77 | at TLSSocket.Readable.push (node:internal/streams/readable:228:10)
78 |  
79 | [Container] 2022/11/16 17:03:28 Command did not exit successfully yarn install --immutable exit status 1

I am using the package cache so I am unsure what network request is being done here. Since corepack seems to be the source I thought it might be fetching yarn. I have yarn@3.2.0 in my cache. First I tried:

corepack enable yarn@3.3.0 --activate

which gave the same error. Then I switched to:

corepack prepare

which worked but fell over at the install step again. I confirmed from the build container that https://registry.yarnpkg.com was responding fine so I don't believe that is the problem.

Is there a way to debug what exactly is happening?

[arcanis]

68 | at ClientRequest. (/usr/local/lib/node_modules/corepack/dist/corepack.js:3933:31)

This line suggests the problem comes from Corepack. It loads the url from here:
https://github.com/nodejs/corepack/blob/main/config.json#L109

If it failed with a 503, it means that Cloudflare returned an error, which (if we assume that Cloudflare doesn't fail) can only happen if there's a connection error when its lambda queries GitHub.

[robinelvin]

Thank you for your prompt reply. I'm still unsure as how to debug this. I don't get this problem anywhere else, only in my CodeBuild environment and it is consistently failing every time. If I could work out where the issue was I might have a chance of raising it with the right team. From what you are saying it seems to pointing to an issue between Cloudflare and the upstream server. Is there a way of logging the full response?
Otherwise I presume I should head over to Corepack and ask there.

[arcanis]

If you have it consistently, and only on CodeBuild, the problem is probably between them and Cloudflare, then. Perhaps Cloudflare blacklisted them?

[robinelvin]

Digging a bit further it is trying to connect to https://repo.yarnpkg.com/tags which appears to be owned by Yarn. It does indeed seem to be a limit imposed by Cloudflare. Since the only way to resolve this is to contact Cloudflare customer support I need to know how to get the Yarn team to sort this. Obviously they and not me are the customer here.

[robinelvin]

So the answer was to create a new VPC with an Elastic IP for the CodeBuild configuration. This has bypassed the blacklisted IP allocated by default and packages are being installed again.

[muya]

Was experiencing a similar error on Gitlab, and switching up the runners helped resolve the issue.