BuildKit is flooding our caches

[thomaseizinger]

I think this came in with the recent interop tests. I've found a mention of "BuildKit" within the "setup docker builx" step: https://github.com/libp2p/rust-libp2p/actions/runs/3994130715/jobs/6851689643#step:8:146

I don't have a solution but the increased number of caches is a problem for us as we are already operating on the given limit of 10GB with our regular CI: https://github.com/libp2p/rust-libp2p/actions/caches

[MarcoPolo]

We can disable the caches for the rust-libp2p CI in the short term. I'll add that as an option to the composite action.

I think we could probably switch to using S3 buckets here and solve this completely as well as give us great caching across branches. Combined with expiring objects could give us the GC we need. @galargh do you think you could help me set up this bucket?

[galargh]

I think we could probably switch to using S3 buckets here and solve this completely as well as give us great caching across branches. Combined with expiring objects could give us the GC we need. @galargh do you think you could help me set up this bucket?

Sure! I created a bucket via https://github.com/pl-strflt/tf-aws-bootstrap/blob/main/libp2p.tf. and shared 2 sets of creds for it with you (read-write and read-only) so that you could copy them to an appropriate, libp2p vault.

I also, in this repository, I set up S3_BUCKET_NAME, S3_AWS_ACCESS_KEY_ID and S3_AWS_ACCESS_KEY_ID_READ_ONLY configuration variables (accessible through ${{ vars.NAME }} in GHA workflows); and S3_SECRET_ACCESS_KEY and S3_SECRET_ACCESS_KEY_READ_ONLY secrets (accessible through ${{ secrets.NAME }} in GHA workflows).

Neither configuration variables nor secrets are accessible from workflows triggered on pull_request. They are accessible from pull_request_target workflows (good accompanying read for this type of workflows - https://securitylab.github.com/research/github-actions-preventing-pwn-requests/).

Finally, s3 buildx cache is not officially released yet. Because of that it requires some additional setup when creating a builder instance. Namely, you need to use a builder image that supports s3 cache. You can do that by providing an image driver opt, e.g. --driver-opt=image=moby/buildkit@sha256:8a45f8c8fcfb0f38e7380d7d9fc728219d2fdf43fd02aee60a2a6723d89abdea. You can see how it all comes together in practice in https://github.com/pl-strflt/tf-aws-gh-runner/blob/main/.github/actions/docker-cache/action.yml#L30.

Not that it matters here but just for completeness of the answer I want to also mention that PL self-hosted GHA runners have no-secrets access to S3 through IAM roles.

[MarcoPolo]

Got this working here: #125

Here's the initial run without caching, builds took 20 minutes: https://github.com/libp2p/test-plans/actions/runs/4068170393/jobs/7006400249

Here's the run with caching, builds took 4 minutes (and this is mostly just fetching the layers): https://github.com/libp2p/test-plans/actions/runs/4068170393/jobs/7006878483

[MarcoPolo]

@galargh could you make an org-level secret var for this bucket please? Maybe call it libp2p container layer cache key (or something similar)

[galargh]

Sure, just give me name-value pairs that you want and I'll make them available on the org level. Do you want the same set of secrets/variables that are currently set on this repo?

[MarcoPolo]

Yes please. Let's call them:

S3_LIBP2P_BUILD_CACHE_AWS_SECRET_ACCESS_KEY
S3_LIBP2P_BUILD_CACHE_AWS_ACCESS_KEY_ID

[galargh]

Added vars:

S3_LIBP2P_BUILD_CACHE_BUCKET_NAME
S3_LIBP2P_BUILD_CACHE_AWS_ACCESS_KEY_ID
S3_LIBP2P_BUILD_CACHE_AWS_ACCESS_KEY_ID_READ_ONLY

Added secrets:

S3_LIBP2P_BUILD_CACHE_AWS_SECRET_ACCESS_KEY
S3_LIBP2P_BUILD_CACHE_AWS_SECRET_ACCESS_KEY_READ_ONLY

I set the visibility to all public repos. Let me know if you prefer it to be only selected ones.