Running a full "build-all.sh" from debuerreotype eventually fails

[tianon]

When proxying all of snapshot.debian.org through squignix, running build-all.sh eventually fails on a somewhat random .deb file (regardless of architecture, the failure is consistent).

An example failure from the debootstrap side:

E: Failed to fetch http://snapshot.debian.org/archive/debian/20180619T000000Z/pool/main/g/gst-libav1.0/gstreamer1.0-libav_1.4.4-2_amd64.deb  Size mismatch

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

The only even semi-related entries in the NGINX log are the following:

2018/06/19 18:06:43 [error] 6#6: *503 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.17.0.3, server: , request: "HEAD /archive/debian/20180619T000000Z/dists/buster-updates/main/binary-amd64/Packages.gz HTTP/1.1", upstream: "http://185.17.185.185:80/archive/debian/20180619T000000Z/dists/buster-updates/main/binary-amd64/Packages.gz", host: "snapshot.debian.org"
2018/06/19 18:06:43 [warn] 6#6: *503 upstream server temporarily disabled while reading response header from upstream, client: 172.17.0.3, server: , request: "HEAD /archive/debian/20180619T000000Z/dists/buster-updates/main/binary-amd64/Packages.gz HTTP/1.1", upstream: "http://185.17.185.185:80/archive/debian/20180619T000000Z/dists/buster-updates/main/binary-amd64/Packages.gz", host: "snapshot.debian.org"
2018/06/19 18:06:44 [error] 6#6: *509 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.17.0.3, server: , request: "HEAD /archive/debian-security/20180619T000000Z/dists/buster/updates/main/binary-amd64/Packages.gz HTTP/1.1", upstream: "http://185.17.185.185:80/archive/debian-security/20180619T000000Z/dists/buster/updates/main/binary-amd64/Packages.gz", host: "snapshot.debian.org"
2018/06/19 18:06:44 [warn] 6#6: *509 upstream server temporarily disabled while reading response header from upstream, client: 172.17.0.3, server: , request: "HEAD /archive/debian-security/20180619T000000Z/dists/buster/updates/main/binary-amd64/Packages.gz HTTP/1.1", upstream: "http://185.17.185.185:80/archive/debian-security/20180619T000000Z/dists/buster/updates/main/binary-amd64/Packages.gz", host: "snapshot.debian.org"

It seems like snapshot.debian.org is having some minor hiccup, and then either max_fails or fail_timeout bites us, but we can't configure those explicitly since we don't (and can't) use a proper upstream block with server entries (since those don't support variables), but that's just a guess. None of the other options in https://nginx.org/en/docs/http/ngx_http_proxy_module.html seem like they'd really help (or hurt) since the only two that seem related default to 0 (proxy_next_upstream_timeout and proxy_next_upstream_tries).

(Filing this issue just to have somewhere to track it -- going to have to put in a workaround to not use squignix for building Debian images for now.)

[tianon]

I've tested now without squignix in the loop and it still fails, so this isn't necessarily specifically a squignix issue, but I think there might be more we can do in squignix itself to help mitigate it, maybe? 😞

[tianon]

Yeah, at the point this happens, we're actually running inside a chroot doing apt-get install (not debootstrap), so we're not hitting Squignix at all, and maybe that's really the problem here. 😄

Might mess with bind-mounting /etc/resolv.conf in debuerreotype to combat this, but regardless there's not actually an issue with Squignix here. 👍

[tianon]

Filed debuerreotype/debuerreotype#39 to improve debuerreotype so it can utilize Squignix more fully. 👍

[tianon]

Detailed issue filed at debuerreotype/debuerreotype#41 (for further digging).