Build is not reproducible: linking order #111

Closed
ChristophBerg opened this Issue Feb 21, 2016 · 5 comments

Comments

Projects
None yet
2 participants
@ChristophBerg
Contributor

ChristophBerg commented Feb 21, 2016

Rebuilding pgbouncer will not always produce the same executable, as the linking order depends on the filesystem order of the object files:

@@ -563,9 +577,9 @@
      CC       lib/usual/safeio.c
      CC       lib/usual/signal.c
      CC       lib/usual/slab.c
-     CC       lib/usual/socket_ntop.c
-     CC       lib/usual/socket_pton.c
      CC       lib/usual/socket.c
+     CC       lib/usual/socket_pton.c
+     CC       lib/usual/socket_ntop.c
      CC       lib/usual/string.c
      CC       lib/usual/strpool.c
      CC       lib/usual/time.c

... which also affects CCLD, but is not visible with V=0. See https://tests.reproducible-builds.org/logdiffs/unstable/armhf/pgbouncer_1.7.1-1.diff.gz for the full build log diff.

Reproducible builds are Debian release goal: https://wiki.debian.org/ReproducibleBuilds

I've been staring at lib/mk/antimake.mk to locate the missing $(sort) which would fix this issue, but the complexity of that file is beyond me. We'd be happy if someone could find a fix for this issue. Thanks!

(There is a second source of unreproducibility which I will report separately.)

@markokr

This comment has been minimized.

Show comment
Hide comment
@markokr

markokr Feb 21, 2016

Contributor

Seems doubtful - most build stuff depends either on sorted wildcards (shell, $(wildcard)) or how stuff is declared in makefile. Really hard to see where FS order comes in.

The diff you posted shows '.' vs '_' difference, plus various noise with LC vars. So please check first if the difference comes from sort order difference in different locale settings.

And what is ReproducibleBuilds attitude to locale management? What part of build is responsible for locale reset?

Contributor

markokr commented Feb 21, 2016

Seems doubtful - most build stuff depends either on sorted wildcards (shell, $(wildcard)) or how stuff is declared in makefile. Really hard to see where FS order comes in.

The diff you posted shows '.' vs '_' difference, plus various noise with LC vars. So please check first if the difference comes from sort order difference in different locale settings.

And what is ReproducibleBuilds attitude to locale management? What part of build is responsible for locale reset?

@ChristophBerg

This comment has been minimized.

Show comment
Hide comment
@ChristophBerg

ChristophBerg Feb 21, 2016

Contributor

Re: Marko Kreen 2016-02-21 pgbouncer/pgbouncer/issues/111/186847962@github.com

Seems doubtful - most build stuff depends either on sorted wildcards (shell, $(wildcard)) or how stuff is declared in makefile. Really hard to see where FS order comes in.

The diff you posted shows '.' vs '_' difference, plus various noise with LC vars. So please check first if the difference comes from sort order difference in different locale settings.

Hi Marko,

I should have provided more context here. The build log diff wasn't
really the best thing to start with, it was just what led me on the
track for the analysis.

The problem is really visible in the binary diff:
https://tests.reproducible-builds.org/dbd/unstable/armhf/pgbouncer_1.7.1-1.diffoscope.html

Interestingly the difference between the first and the second build is
only in the debug symbols (and in turn in the build-id placed into the
pgbouncer executable/.deb file). The "readelf --wide --symbols {}"
output suggests that the difference is due to the socket*.c files
having a different order.

And what is ReproducibleBuilds attitude to locale management? What part of build is responsible for locale reset?

The idea is that the build result is independent of the locale
selected (that is why one of the builds in the build log diff was
running in French). They vary various parameters in the build
environment there, build things twice, and compare the result.

One particular item varied is the order of items returned by readdir()
(via "disorderfs"), which seems to be the culprit here.

NB: $(wildcard) doesn't sort (while shell globs are sorted by
default).

Christoph

cb@df7cb.de | http://www.df7cb.de/

Contributor

ChristophBerg commented Feb 21, 2016

Re: Marko Kreen 2016-02-21 pgbouncer/pgbouncer/issues/111/186847962@github.com

Seems doubtful - most build stuff depends either on sorted wildcards (shell, $(wildcard)) or how stuff is declared in makefile. Really hard to see where FS order comes in.

The diff you posted shows '.' vs '_' difference, plus various noise with LC vars. So please check first if the difference comes from sort order difference in different locale settings.

Hi Marko,

I should have provided more context here. The build log diff wasn't
really the best thing to start with, it was just what led me on the
track for the analysis.

The problem is really visible in the binary diff:
https://tests.reproducible-builds.org/dbd/unstable/armhf/pgbouncer_1.7.1-1.diffoscope.html

Interestingly the difference between the first and the second build is
only in the debug symbols (and in turn in the build-id placed into the
pgbouncer executable/.deb file). The "readelf --wide --symbols {}"
output suggests that the difference is due to the socket*.c files
having a different order.

And what is ReproducibleBuilds attitude to locale management? What part of build is responsible for locale reset?

The idea is that the build result is independent of the locale
selected (that is why one of the builds in the build log diff was
running in French). They vary various parameters in the build
environment there, build things twice, and compare the result.

One particular item varied is the order of items returned by readdir()
(via "disorderfs"), which seems to be the culprit here.

NB: $(wildcard) doesn't sort (while shell globs are sorted by
default).

Christoph

cb@df7cb.de | http://www.df7cb.de/

@markokr

This comment has been minimized.

Show comment
Hide comment
@markokr

markokr Feb 21, 2016

Contributor

Hadn't realized about $(wildcard). I'm still on 3.81...

Please test if this commit in libusual fixes it.

Contributor

markokr commented Feb 21, 2016

Hadn't realized about $(wildcard). I'm still on 3.81...

Please test if this commit in libusual fixes it.

@markokr

This comment has been minimized.

Show comment
Hide comment
@markokr

markokr Feb 22, 2016

Contributor

Seems github auto-closed it. Please reopen if problem persist.

Contributor

markokr commented Feb 22, 2016

Seems github auto-closed it. Please reopen if problem persist.

@ChristophBerg

This comment has been minimized.

Show comment
Hide comment
@ChristophBerg

ChristophBerg Feb 25, 2016

Contributor

The build is reproducible now. Thanks!

Contributor

ChristophBerg commented Feb 25, 2016

The build is reproducible now. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment