feat: runs-on rollout part 1 (DX-365)

[erikburt]

Initial rollout for runs-on self hosted runners.

After this PR is merged, only PRs with runs-on label will use self-hosted runners. Incremental rollout will start happening after this PR is merged.

This does change the build cache key but should have minimal to no effect for existing PRs once the proper caches on develop are populated.

Changes

1. Update runner labels to use inline notation for runners (for now)
2. Update the build cache to use a short sha in the key
   • The idea behind this is that our build caches can grow stale but we will never upsert/overwrite them in their current key format. With the ability to have unlimited cache with s3 self-hosted runners, we can effectively upsert / overwrite our caches everytime a new ref is pushed.
   • Currently, PRs which make multiple changes over time, the build cache is still synced to the first time the workflow was run.
3. Add a 20 minute timeout to our unit tests. Anything longer than 20 minutes, means something is likely hanging and should fail.
   • This is increased for scheduled runs because race tests are extended during these.
4. Add opt-in / opt-out labels for self-hosted runners
   • If a PR now adds runs-on it will use self-hosted runners
   • When we start rolling this out across PRs, if the PR has runs-on-opt-out label it will use Github runners
5. Do not run sonarqube scan if the workflow was cancelled.
   • This job was configured with an always() which means it would still run even if the workflow was cancelled, and it was annoying.

Notes

There are some problems with the self-hosted runners. Although they are rare its possible:

1. We do use spot instances when possible and spot instances can be interrupted, so sometimes a job will fail.
   • We do have automatic retries on, so if a job is interrupted by a spot instance termination, it will be automatically retried (this creates problem - see point 2)
   • I have seen it a few times in testing. If this is happening too often, then there are also ways to reduce the occurrence.
   • ie. Only use on-demand instances for merge queue so it won't interrupt crucial workflows.
2. Automatic retries will interrupt new workflow runs:
   • If a job is cancelled due to a spot instance termination, and another commit is pushed, then runs-on will retry the already failed job, which will cause the newer workflow to fail.
   • This is a very rare edge case that shouldn't pop up too much, if it does there are ways to disable automatic retries.

Testing

I have run a ton of tests using this PR / branch:

• https://github.com/smartcontractkit/chainlink/actions/runs/14227410891/job/39870270989?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14227773342/job/39871366996?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14228065115/job/39872249724?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14228352375/job/39873206071?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14228563782/job/39874763941?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14229343564/job/39876406784?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14229528429/job/39877094165?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14229918030/job/39878180631?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14230168886/job/39878949951?pr=17026
• https://github.com/smartcontractkit/chainlink/actions/runs/14230478702/job/39879954354?pr=17026

---

DX-365

[cl-sonarqube-production]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[sebawo]

@kalverra can you use your tool to compare execution times between hosted runners and self-hosted runners + unlimited cache?

[sebawo]

@erikburt do i understand correctly that in this last run (https://github.com/smartcontractkit/chainlink/actions/runs/14230478702/job/39879953171?pr=17026) we leveraged the cache the most and this is why Core Tests (go_core_tests) run in 5min 17 sec and in the first run (https://github.com/smartcontractkit/chainlink/actions/runs/14227410891/job/39870270989?pr=17026) it was 16min 26 sec?

[erikburt]

@erikburt do i understand correctly that in this last run (smartcontractkit/chainlink/actions/runs/14230478702/job/39879953171?pr=17026) we leveraged the cache the most and this is why Core Tests (go_core_tests) run in 5min 17 sec and in the first run (smartcontractkit/chainlink/actions/runs/14227410891/job/39870270989?pr=17026) it was 16min 26 sec?

Yes - 1st run was no cache, but was also abnormally slow. There was probably a hanging test or something that had to be retried.

ccip deployment tests have seen almost no runtime improvement, and I don't understand why. I plan on digging into this more.