OCI workflow overloads GHA cache

[aaronmondal]

This is not reproducible locally and seems to be caused by our aggressive use of the GHA cache which makes an estimated ~400 queries to the GHA cache. GHA appears to just close the connection at some point and that causes the workflow to break.

As temporary workaround we can disable caching for the workflow, but we'll need to find a better solution as caching has a significant impact on this workflow's runtime.

[adam-singer]

GHA generally ok for simple things, at some point it just doesn't scale with workloads/size/complexity. https://buildkite.com/ tends to be a popular one among more complicated setups/verifications and github repos.

[aaronmondal]

In this case we'd need a nix-specific cache. We don't store the entire cache as a single blob but instead cache each derivation individually: https://github.com/TraceMachina/nativelink/actions/caches For most builds this is fine, but our rust setup in nix caches each crate individually.

AFAIK the only commercial offering for a nix cache is https://cachix.org. We can also roll our own cache since we have the infrastructure for it.

[aaronmondal]

We could deploy https://github.com/zhaofengli/attic