ci: Make buildkite agents stateless and unique per job

[dan-mckean]

Less pain releasing: We schedule buildkite to run on top of k8s, and they are stateful, so there are leftovers from previous builds, sometimes this results in a random failure. Make the agents stateless, make the agents unique per job. There is a bias to NOT extend the build times

[github-actions]

Heads up @davejrt @ggilmore @daxmc99 - the "team/distribution" label was applied to this issue.

[jhchabran]

@davejrt I assigned myself there as well, this sounds rather big otherwise as a one person task.

[jhchabran]

@davejrt quick heads up, I did a round of tests with https://verdaccio.org/ and https://github.com/stackdumper/npm-cache-proxy and it did not yield significant improvements, in the case of Verdaccio, it even came with errors fetching some packages was overall slower than without any cache.

My general sentiment, is that https://github.com/gencer/cache-buildkite-plugin is a more sensible approach and is generic enough that anyone could introduce a caching mechanism when they need it within their build steps. We could use it for asdf for example, using the tools-version file checksum as the cache key.

I'd like to run some tests involving it, but I think I'll need some help regarding how to set up a GCS bucket from Terraform or use a PVC.