CI: configure apt-cacher / mirror for packages (or other solutions)

[thaJeztah]

Also related: #39512

CI is currently configured to use the cdn-fastly.deb.debian.org mirror. This was primarily done to fix flakiness in CI (where the upstream package repositories were down);

• [Build process] Error fetching deb #21512 [Build process] Error fetching deb
• Introduce dockercore/buildbase x64 base image #20665 Introduce dockercore/buildbase x64 base image
• Allow specifying apt mirror in dockerfile #21590 Allow specifying apt mirror in dockerfile
• allow replacing httpredir mirror for arm #28640 allow replacing httpredir mirror for arm

We should:

Investigate if we still should specify the mirror manually

From the discussion in

I've got high hopes for http://deb.debian.org, but it's currently stretch-only, and isn't supported by debootstrap (yet), so we'd have to hard-code cdn-fastly.deb.debian.org for now, and I'm not sure how well-supported that'll stay without an official announcement that it ought to be used on debian-devel-announce. 😞

Looking at the page at http://httpredir.debian.org, deb.debian.org already redirects to a mirror

If you hit the server behind deb.debian.org directly, either because you use an older apt or because you use a HTTP proxy that does not support SRV records, your requests will get HTTP redirected to one of the CDN instances. If you want to avoid the redirects, you can pick one instance directly. For instance, this also works in your sources.list:

deb http://cdn-fastly.deb.debian.org/debian stable main
deb http://cdn-fastly.deb.debian.org/debian-security stable/updates main

The redirection service is also available on HTTPS, so with the apt-transport-https package installed, you can use:

deb https://deb.debian.org/debian stable main
deb https://deb.debian.org/debian-security stable/updates main

This service is sponsored by Fastly and Amazon CloudFront.

• Is dn-fastly.deb.debian.org the only mirror it redirects to?
• Are there other mirrors that are faster for the machines we run on?
  • i.e., for the AWS machines, is there a local/faster mirror that we currently exclude by manually overriding?
  • the s390x/PowerPC/Windows machines run on different infrastructure; is this the optimum mirror for those?

Investigate if setting up apt-cacher would improve performance

We should investigate if setting up a apt-cacher / mirror for the packages we use improves speed; see comments on #21512 (comment)

Another idea is to install our own apt-cacher, so that we can can cache the packages we use close to us, and limit the impact of a bad mirror. see https://help.ubuntu.com/community/Apt-Cacher-Server for more details.
..

https://docs.docker.com/engine/examples/apt-cacher-ng/

We'll have to profile the docker build steps to see how long they take (switching to BuildKit for the builds might give us more details about the builds through opentracing (moby/buildkit#255), and we could collect metrics about that?)

Investigate using BuildKit (distributed) cache

Perhaps instead of optimising at the package-manager level, taking advantage of BuildKit's improved caching might be a better solution. #20665 (Introduce dockercore/buildbase x64 base image) turned out to not be a good solution at the time, but things may have changed with BuildKit.

[thaJeztah]

/cc @tianon @andrewhsu @tonistiigi @AkihiroSuda

[tianon]

I would say using the default deb.debian.org should usually be pretty reasonable now -- it's much more stable than httpredir was.

I used to use apt-cacher-ng extensively, but found it was not as active as I needed it to be at expiring cache, so I currently just use NGINX configured to act like Squid as a reverse proxy in front of deb.debian.org and it's been working pretty well (using NGINX because I understand how to configure NGINX better than the strange ways Squid wants to be configured, but any tool should do the trick).

To apply it, I transparently hijack the DNS of all my containers and point deb.debian.org directly at my reverse proxy (so that all builds benefit instead of having to manually modify every build I make to point at my proxy, which really wouldn't fly for 90% of the builds that I do).

Edit: in case you're interested in the specifics of my NGINX configuration, that's at https://github.com/tianon/squignix.