Speed up image builds with Docker cache mounts

[zanieb]

Inspired by the slightly disturbing discussion at rust-lang/cargo#2644

[zanieb]

Oh sigh cache mounts don't work in GitHub Actions:

• Allow controlling cache mounts storage location moby/buildkit#1512
• --mount=type=cache not caching properly? docker/build-push-action#716
• mount=type=cache more in-depth explanation? moby/buildkit#1673

Might need to cache /var/lib/buildkitper moby/buildkit#1512 (comment)
or do something like:

• https://github.com/dashevo/gh-action-cache-buildkit-state
• Allow controlling cache mounts storage location moby/buildkit#1512 (comment)
• https://github.com/overmindtech/buildkit-cache-dance
• https://depot.dev/

[konstin]

What i like to do is to have an empty crate with nothing but the Cargo.toml, Cargo.lock and dummy lib.rs/main.rs, build that, then add the source and build again. The first build gets cached and when the deps didn't change the second build runs with all deps already built (https://github.com/PyO3/maturin/blob/3492c978e9a3031aaa60f3c7051990f2c621fd73/Dockerfile#L17-L25).

[zanieb]

That makes sense although it feels like more of a hack; several comments suggest that in rust-lang/cargo#2644.

The cache mounts seem like a total pain to do in CI.