Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trying to build an "oldstable" ("jessie") tarball fails trying to download "gstreamer1.0-libav" (which nothing depends on and shouldn't be pulled in at all) #41

Closed
tianon opened this issue Jun 22, 2018 · 10 comments

Comments

@tianon
Copy link
Collaborator

tianon commented Jun 22, 2018

Somewhere between 20180426 and today, something has changed such that the following failure is 100% reproducible while trying to do something like ./build.sh --codename-copy output oldstable 'today' (where today can be replaced by any recent timestamp).

I've tried a number of things such as #39 which I thought would help because the error looks like a download failure so I was hoping my Squignix hack could help paper over the failure, but looking into it it's much more nefarious than that (see tianon/squignix#2 for where I initially filed a note about this). Here's the failure output with a bunch of context that I'll explain following:

+ debuerreotype-apt-get rootfs-sbuild install -y --no-install-recommends build-essential fakeroot
Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  binutils bzip2 cpp cpp-4.9 dpkg-dev g++ g++-4.9 gcc gcc-4.9 libasan1
  libatomic1 libc-dev-bin libc6-dev libcilkrts5 libcloog-isl4 libdpkg-perl
  libfakeroot libgcc-4.9-dev libgdbm3 libgmp10 libgomp1 libisl10 libitm1
  liblsan0 libmpc3 libmpfr4 libquadmath0 libstdc++-4.9-dev libtimedate-perl
  libtsan0 libubsan0 linux-libc-dev make patch perl perl-modules xz-utils
Suggested packages:
  binutils-doc bzip2-doc cpp-doc gcc-4.9-locales debian-keyring g++-multilib
  g++-4.9-multilib gcc-4.9-doc libstdc++6-4.9-dbg gcc-multilib manpages-dev
  autoconf automake libtool flex bison gdb gcc-doc gcc-4.9-multilib
  libgcc1-dbg libgomp1-dbg libitm1-dbg libatomic1-dbg libasan1-dbg
  liblsan0-dbg libtsan0-dbg libubsan0-dbg libcilkrts5-dbg libquadmath0-dbg
  glibc-doc libstdc++-4.9-doc make-doc ed diffutils-doc perl-doc
  libterm-readline-gnu-perl libterm-readline-perl-perl libb-lint-perl
  libcpanplus-dist-build-perl libcpanplus-perl libfile-checktree-perl
  liblog-message-simple-perl liblog-message-perl libobject-accessor-perl
Recommended packages:
  libalgorithm-merge-perl libfile-fcntllock-perl netbase rename
  libarchive-extract-perl libmodule-pluggable-perl libpod-latex-perl
  libterm-ui-perl libtext-soundex-perl libcgi-pm-perl libmodule-build-perl
  libpackage-constants-perl
The following NEW packages will be installed:
  binutils build-essential bzip2 cpp cpp-4.9 dpkg-dev fakeroot g++ g++-4.9 gcc
  gcc-4.9 libasan1 libatomic1 libc-dev-bin libc6-dev libcilkrts5 libcloog-isl4
  libdpkg-perl libfakeroot libgcc-4.9-dev libgdbm3 libgmp10 libgomp1 libisl10
  libitm1 liblsan0 libmpc3 libmpfr4 libquadmath0 libstdc++-4.9-dev
  libtimedate-perl libtsan0 libubsan0 linux-libc-dev make patch perl
  perl-modules xz-utils
0 upgraded, 39 newly installed, 0 to remove and 0 not upgraded.
Need to get 48.4 MB of archives.
After this operation, 171 MB of additional disk space will be used.
Get:1 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libgdbm3 amd64 1.8.3-13.1 [30.0 kB]
Get:2 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main perl-modules all 5.20.2-3+deb8u11 [2547 kB]
Get:3 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main perl amd64 5.20.2-3+deb8u11 [2642 kB]
Get:4 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libasan1 amd64 4.9.2-10+deb8u1 [195 kB]
Get:5 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libatomic1 amd64 4.9.2-10+deb8u1 [9014 B]
Get:6 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libcilkrts5 amd64 4.9.2-10+deb8u1 [40.1 kB]
Get:7 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libgmp10 amd64 2:6.0.0+dfsg-6 [253 kB]
Get:8 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libisl10 amd64 0.12.2-2 [440 kB]
Get:9 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libcloog-isl4 amd64 0.18.2-1+b2 [61.8 kB]
Get:10 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libgomp1 amd64 4.9.2-10+deb8u1 [37.8 kB]
Get:11 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libitm1 amd64 4.9.2-10+deb8u1 [29.3 kB]
Get:12 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main liblsan0 amd64 4.9.2-10+deb8u1 [92.7 kB]
Get:13 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libmpfr4 amd64 3.1.2-2 [527 kB]
Get:14 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libquadmath0 amd64 4.9.2-10+deb8u1 [129 kB]
Get:15 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libtsan0 amd64 4.9.2-10+deb8u1 [212 kB]
Get:16 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libubsan0 amd64 4.9.2-10+deb8u1 [82.4 kB]
Get:17 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libmpc3 amd64 1.0.2-1 [39.3 kB]
Get:18 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main bzip2 amd64 1.0.6-7+b3 [46.9 kB]
Get:19 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main patch amd64 2.7.5-1 [109 kB]
Get:20 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main xz-utils amd64 5.1.1alpha+20120614-2+b3 [221 kB]
Get:21 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main binutils amd64 2.25-5+deb8u1 [3496 kB]
Get:22 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libc-dev-bin amd64 2.19-18+deb8u10 [238 kB]
Get:23 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main linux-libc-dev amd64 3.16.56-1+deb8u1 [1096 kB]
Get:24 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libc6-dev amd64 2.19-18+deb8u10 [2003 kB]
Get:25 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main cpp-4.9 amd64 4.9.2-10+deb8u1 [5002 kB]
Get:26 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main cpp amd64 4:4.9.2-2 [17.3 kB]
Get:27 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libgcc-4.9-dev amd64 4.9.2-10+deb8u1 [2066 kB]
Get:28 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main gcc-4.9 amd64 4.9.2-10+deb8u1 [5184 kB]
Get:29 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main gcc amd64 4:4.9.2-2 [5136 B]
Get:30 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main libstdc++-4.9-dev amd64 4.9.2-10+deb8u1 [1120 kB]
Get:31 http://snapshot.debian.org/archive/debian-security/20180622T000000Z/ oldstable/updates/main g++-4.9 amd64 4.9.2-10+deb8u1 [17.3 MB]
Get:32 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main g++ amd64 4:4.9.2-2 [1530 B]
Get:33 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main make amd64 4.0-8.1 [349 kB]
Get:34 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libtimedate-perl all 2.3000-2 [42.2 kB]
Get:35 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libdpkg-perl all 1.17.27 [1075 kB]
Get:36 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main dpkg-dev all 1.17.27 [1548 kB]
Get:37 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main build-essential amd64 11.7 [7114 B]
Get:38 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main libfakeroot amd64 1.20.2-1 [44.7 kB]
Get:39 http://snapshot.debian.org/archive/debian/20180622T000000Z/ oldstable/main fakeroot amd64 1.20.2-1 [84.7 kB]
Fetched 48.1 MB in 1min 6s (720 kB/s)
E: Failed to fetch http://snapshot.debian.org/archive/debian/20180622T000000Z/pool/main/g/gst-libav1.0/gstreamer1.0-libav_1.4.4-2_amd64.deb  Size mismatch

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

If you're more in-tune than I was the first 20 or so times I looked at this output, you might notice that not only is gstreamer1.0-libav not one of the 39 packages APT tried to download, it's also not listed as a dependency of anything being installed (and this is happening in a pretty standard/faithful minbase chroot -- basically just debuerreotype-init, dist-upgrade, followed by debuerreotype-minimizing-config), so uh, what??

I've tried re-ordering the lines in debuerreotype-gen-sources-list to match the order of https://salsa.debian.org/installer-team/apt-setup/tree/master/generators instead of https://wiki.debian.org/SourcesList#Example_sources.list (silly, but figure it was an easy change to rule out right off).

I've tried reproducing by using the current debian:jessie-slim Docker image and updating its sources.list to use the same exact snapshot.debian.org links (identical sources.list content), no avail.

I've tried removing the tmpfs that we build the rootfs on top of (just in case), no avail.

This is 100% reproducible at this point on several AWS hosts and my local machine, so I don't think it's environmental either.

My next attempt is going to be essentially doing a binary search of snapshot.debian.org timestamps to try and narrow down what's changed in the archive between the known-good and current timestamps to try and glean more clues. 😞

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

Just verified ./build.sh --codename-copy output oldstable '2018-04-26 00:00:00', and it fails too, so this is looking more like something that's changed in either Stretch or debuerreotype! 😞

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

It's confirmed!!!! #37 is the culprit. Apparently APT in Jessie cannot actually handle that (so we'll need to update that to be Stretch and later instead).

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

Also confirmed that ./build.sh --codename-copy output oldstable '2018-01-01 00:00:00' works flawlessly, which IMO makes this even more bizarre, but I don't think I want to really dig into it much more.

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

(Thinking of adding today 00:00:00 builds to Travis that don't verify a particular hash just to notify me if something like this happens again but doesn't affect the older snapshot we're currently testing.)

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

Trying to help narrow this down, I've walked backwards to 2018-03-01 00:00:00, where the failure happens sooner (during dist-upgrade), and is a more hilarious package:

+ debuerreotype-apt-get rootfs dist-upgrade -yqq
E: Failed to fetch http://snapshot.debian.org/archive/debian-security/20180301T000000Z/pool/updates/main/libr/libreoffice/libreoffice-core_4.3.3-2+deb8u10_amd64.deb  Size mismatch

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

(We definitely don't ever install anything that might pull in libreoffice-core. 😆)

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

So, I'm starting to think this probably isn't something that changed in a package, but probably more likely something where one of the Packages files has changed size just right to where the xz code in APT is having a bizarre overrun or underrun bug? Still digging to try and get better insight.

To recap: I'm able to reproduce by doing the following in a debian:jessie container (debian:jessie-20180426 to be more specific, but I don't think that's terribly relevant):

$ sed -i '/CompressionTypes/d' /etc/apt/apt.conf.d/docker-gzip-indexes
$ apt-get update
$ apt-get dist-upgrade # not sure this step is necessary
$ apt-get install --no-install-recommends build-essential fakeroot

Here's the results of my rough binary search to narrow down further exactly when things changed enough to cause this behavior: (doing ./build.sh --codename-copy output oldstable 'DATE 00:00:00')

  • 2018-02-01: ✔️ (build succeeds)
  • 2018-03-01: ❌ (E: Failed to fetch http://snapshot.debian.org/archive/debian-security/20180301T000000Z/pool/updates/main/libr/libreoffice/libreoffice-core_4.3.3-2+deb8u10_amd64.deb Size mismatch)
  • 2018-02-14: ✔️
  • 2018-02-21: ❌ (E: Failed to fetch http://snapshot.debian.org/archive/debian-security/20180221T000000Z/pool/updates/main/libr/libreoffice/libreoffice-common_4.3.3-2+deb8u10_all.deb Size mismatch)
  • 2018-02-18: ❌ (E: Failed to fetch http://snapshot.debian.org/archive/debian-security/20180218T000000Z/pool/updates/main/libr/libreoffice/libreoffice-common_4.3.3-2+deb8u10_all.deb Size mismatch)
  • 2018-02-16: ✔️
  • 2018-02-17: ✔️

So, something changed between 2018-02-17 and 2018-02-18. 👍

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

Here's the request logs of the debian-security archive contents for my 2018-02-18 attempt:

MISS 200 HEAD "http://snapshot.debian.org:80/archive/debian-security/20180218T000000Z/dists/oldstable/updates/main/binary-amd64/Packages.gz" HTTP/1.1 [22/Jun/2018:13:16:51 -0700] 172.18.0.27 "Wget/1.18 (linux-gnu)"
MISS 200 GET "http://snapshot.debian.org:80/archive/debian-security/20180218T000000Z/dists/oldstable/updates/InRelease" HTTP/1.1 [22/Jun/2018:13:17:07 -0700] 172.18.0.27 "Debian APT-HTTP/1.3 (1.0.9.8.4)"
MISS 200 GET "http://snapshot.debian.org:80/archive/debian-security/20180218T000000Z/dists/oldstable/updates/main/binary-amd64/Packages.bz2" HTTP/1.1 [22/Jun/2018:13:17:15 -0700] 172.18.0.27 "Debian APT-HTTP/1.3 (1.0.9.8.4)"
MISS 200 GET "http://snapshot.debian.org:80/archive/debian-security/20180218T000000Z/pool/updates/main/g/gcc-4.9/gcc-4.9-base_4.9.2-10%2bdeb8u1_amd64.deb" HTTP/1.1 [22/Jun/2018:13:18:04 -0700] 172.18.0.27 "Debian APT-HTTP/1.3 (1.0.9.8.4)"
MISS 200 GET "http://snapshot.debian.org:80/archive/debian-security/20180218T000000Z/pool/updates/main/g/gcc-4.9/libstdc%2b%2b6_4.9.2-10%2bdeb8u1_amd64.deb" HTTP/1.1 [22/Jun/2018:13:18:12 -0700] 172.18.0.27 "Debian APT-HTTP/1.3 (1.0.9.8.4)"
MISS 200 GET "http://snapshot.debian.org:80/archive/debian-security/20180218T000000Z/pool/updates/main/g/gcc-4.9/libgcc1_4.9.2-10%2bdeb8u1_amd64.deb" HTTP/1.1 [22/Jun/2018:13:18:13 -0700] 172.18.0.27 "Debian APT-HTTP/1.3 (1.0.9.8.4)"
MISS 200 GET "http://snapshot.debian.org:80/archive/debian-security/20180218T000000Z/pool/updates/main/libr/libreoffice/libreoffice-common_4.3.3-2%2bdeb8u10_all.deb" HTTP/1.1 [22/Jun/2018:13:18:50 -0700] 172.18.0.27 "Debian APT-HTTP/1.3 (1.0.9.8.4)"

Interesting that it does a HEAD on dists/oldstable/updates/main/binary-amd64/Packages.gz (emphasis on the gz), then does a GET on dists/oldstable/updates/InRelease followed by a GET on dists/oldstable/updates/main/binary-amd64/Packages.bz2 (emphasis on the bz2), but maybe that's entirely normal.

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

One major change between 2018-02-17 and 2018-02-18 is that we got a security update to src:gcc-4.9 (4.9.2-10+deb8u1), which might be related?

(Looking at a diff between http://snapshot.debian.org/archive/debian-security/20180217T000000Z/dists/oldstable/updates/main/binary-amd64/Packages.bz2 and http://snapshot.debian.org/archive/debian-security/20180218T000000Z/dists/oldstable/updates/main/binary-amd64/Packages.bz2 to try and find clues.)

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

As expected, http://snapshot.debian.org/archive/debian/20180217T000000Z/dists/oldstable/main/binary-amd64/Packages.xz and http://snapshot.debian.org/archive/debian/20180218T000000Z/dists/oldstable/main/binary-amd64/Packages.xz are identical (which is good).

Ditto with http://snapshot.debian.org/archive/debian/20180217T000000Z/dists/oldstable-updates/main/binary-amd64/Packages.xz and http://snapshot.debian.org/archive/debian/20180218T000000Z/dists/oldstable-updates/main/binary-amd64/Packages.xz.

That src:gcc-4.9 update is the only change in this entire time period:

$ diff -u <(curl -fsSL 'http://snapshot.debian.org/archive/debian-security/20180217T000000Z/dists/oldstable/updates/main/binary-amd64/Packages.bz2' | bzcat) <(curl -fsSL 'http://snapshot.debian.org/archive/debian-security/20180218T000000Z/dists/oldstable/updates/main/binary-amd64/Packages.bz2' | bzcat) | grep 'Source:' | sort -u
+Source: gcc-4.9
+Source: gcc-4.9 (4.9.2-10+deb8u1)
 Source: gdk-pixbuf

@tianon
Copy link
Collaborator Author

tianon commented Jun 22, 2018

Giving up on digging further to figure out the root cause; filed #42 to patch around the issue instead (especially since Jessie transitions to LTS this weekend).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant