`test-rackspace-win2022_vs2022-x64-6` systematic build timeout for `win-vs2022-arm64`

[targos]

See https://ci.nodejs.org/computer/test-rackspace-win2022_vs2022-x64-6/builds

All win-vs2022-arm64 systematically fail with:

00:56:43 Cross-compilation to ARM64 detected. We'll use the x64 Node executable for license2rtf.
01:56:43 Build timed out (after 60 minutes). Marking the build as failed.
01:56:44 Build was aborted

Examples:
https://ci.nodejs.org/job/node-compile-windows/nodes=win-vs2022-arm64/52608/console
https://ci.nodejs.org/job/node-compile-windows/nodes=win-vs2022-arm64/52601/console
https://ci.nodejs.org/job/node-compile-windows/nodes=win-vs2022-arm64/52597/console

Other machines don't have this problem so I removed the win-vs2022-arm64 label from that one in https://ci.nodejs.org/computer/test-rackspace-win2022_vs2022-x64-6/configure to prevent CI failures.

[joyeecheung]

nodejs/reliability#676 indicates that this happens on test-rackspace-win2022_vs2022-x64-1 and test-rackspace-win2022_vs2022-x64-2 too, maybe less frequent though

[richardlau]

00:56:43 Cross-compilation to ARM64 detected. We'll use the x64 Node executable for license2rtf.

This message is immediately before an attempt is made to download a x64 Node.js Windows binary (to convert the license to RTF). #3477 (comment) says that on the test CI that there should be a caching mechanism to prevent the builds getting stuck here.

[targos]

Two win2019 machines are also affected:
https://ci.nodejs.org/computer/test-rackspace-win2019_vs2019-x64-1/builds
https://ci.nodejs.org/computer/test-rackspace-win2019_vs2019-x64-2/builds

[StefanStojanovic]

I took a look. There is indeed a cache there but it can get corrupted (that's what happened here). I'll improve the logic to prohibit this. There is also an occasional ongoing problem with release CI where nodes get stuck trying to download the x64 node.exe for days (no caching there). I'll try to make a standardized approach that will work in both CI environments.

[targos]

Label added back to https://ci.nodejs.org/computer/test-rackspace-win2022_vs2022-x64-6/configure

[targos]

@StefanStojanovic It seems the bug is still here: https://ci.nodejs.org/job/node-compile-windows/53041/nodes=win-vs2022-arm64/console

[StefanStojanovic]

Thanks for letting me know @targos! This is a different bug, but it represents the same. Caching only works well if it can download the x64 binary, in runs after you reenabled the machine I've seen this when downloading node.exe:

curl: (18) transfer closed with 69583820 bytes remaining to read

The machine didn't have a cached node.exe and in one run compilation failed. The new caching mechanism improves on the old one in a way that it doesn't overwrite a valid binary with an invalid one. Since this machine didn't have a valid binary to begin with, failure was a possibility. Now I've set it up correctly and checked other machines for invalid cache. Since now everything is as expected, the new caching mechanism should make sure it stays like that by never invalidating node.exe.

I'll be monitoring this in the following weeks and hopefully, this will not happen again.