Build is not reproducible: linking order

[df7cb]

Rebuilding pgbouncer will not always produce the same executable, as the linking order depends on the filesystem order of the object files:

@@ -563,9 +577,9 @@
      CC       lib/usual/safeio.c
      CC       lib/usual/signal.c
      CC       lib/usual/slab.c
-     CC       lib/usual/socket_ntop.c
-     CC       lib/usual/socket_pton.c
      CC       lib/usual/socket.c
+     CC       lib/usual/socket_pton.c
+     CC       lib/usual/socket_ntop.c
      CC       lib/usual/string.c
      CC       lib/usual/strpool.c
      CC       lib/usual/time.c

... which also affects CCLD, but is not visible with V=0. See https://tests.reproducible-builds.org/logdiffs/unstable/armhf/pgbouncer_1.7.1-1.diff.gz for the full build log diff.

Reproducible builds are Debian release goal: https://wiki.debian.org/ReproducibleBuilds

I've been staring at lib/mk/antimake.mk to locate the missing $(sort) which would fix this issue, but the complexity of that file is beyond me. We'd be happy if someone could find a fix for this issue. Thanks!

(There is a second source of unreproducibility which I will report separately.)

[markokr]

Seems doubtful - most build stuff depends either on sorted wildcards (shell, $(wildcard)) or how stuff is declared in makefile. Really hard to see where FS order comes in.

The diff you posted shows '.' vs '_' difference, plus various noise with LC vars. So please check first if the difference comes from sort order difference in different locale settings.

And what is ReproducibleBuilds attitude to locale management? What part of build is responsible for locale reset?

[df7cb]

Re: Marko Kreen 2016-02-21 pgbouncer/pgbouncer/issues/111/186847962@github.com

Seems doubtful - most build stuff depends either on sorted wildcards (shell, $(wildcard)) or how stuff is declared in makefile. Really hard to see where FS order comes in.

The diff you posted shows '.' vs '_' difference, plus various noise with LC vars. So please check first if the difference comes from sort order difference in different locale settings.

Hi Marko,

I should have provided more context here. The build log diff wasn't
really the best thing to start with, it was just what led me on the
track for the analysis.

The problem is really visible in the binary diff:
https://tests.reproducible-builds.org/dbd/unstable/armhf/pgbouncer_1.7.1-1.diffoscope.html

Interestingly the difference between the first and the second build is
only in the debug symbols (and in turn in the build-id placed into the
pgbouncer executable/.deb file). The "readelf --wide --symbols {}"
output suggests that the difference is due to the socket*.c files
having a different order.

And what is ReproducibleBuilds attitude to locale management? What part of build is responsible for locale reset?

The idea is that the build result is independent of the locale
selected (that is why one of the builds in the build log diff was
running in French). They vary various parameters in the build
environment there, build things twice, and compare the result.

One particular item varied is the order of items returned by readdir()
(via "disorderfs"), which seems to be the culprit here.

NB: $(wildcard) doesn't sort (while shell globs are sorted by
default).

Christoph

cb@df7cb.de | http://www.df7cb.de/

[markokr]

Hadn't realized about $(wildcard). I'm still on 3.81...

Please test if this commit in libusual fixes it.

[markokr]

Seems github auto-closed it. Please reopen if problem persist.

[df7cb]

The build is reproducible now. Thanks!