mono builds are nondeterministic

[bmwiedemann]

While working on reproducible builds for openSUSE, I found that
.exe and .dll files produced by mono differ in every build,
even when reducing environmental differences to a minimum.

Originally filed at https://bugzilla.suse.com/show_bug.cgi?id=1141502

Steps to Reproduce

0. on openSUSE, install required packages zypper in osc build build-compare mono-devel
1. build an openSUSE C# package with mono twice osc checkout openSUSE:Factory/smuxi && cd $_ && for n in 1 2 ; do osc build --noservice --keep-pkgs=rpms-$n ; done
2. compare results /usr/lib/build/pkg-diff.sh rpms-[12]/smuxi-engine-1*noarch.rpm

Current Behavior

.exe and .dll files differ for every build

--- old /usr/lib/smuxi/smuxi-engine.dll (monodis)
+++ new /usr/lib/smuxi/smuxi-engine.dll (monodis)
@@ -52,7 +52,7 @@
 }
 .assembly extern ServiceStack.Text
 {
-  .ver 3:5:7:21894
+  .ver 3:5:7:21932
 }
 .assembly extern System.Xml
 {

Expected Behavior

It should be possible to get bit-identical build results later on another machine.
See https://reproducible-builds.org/ for why this matters.

On which platforms did you notice this

[ ] macOS
[X] Linux
[ ] Windows

Version Used:

mono-devel-6.8.0

The only thing I could find on the topic was https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=851809 and that is probably unrelated.

[akoeplinger]

You need to make sure to pass the -deterministic option to the C# compiler (csc) to make its output deterministic, however what you see in your case is that you're compiling against different versions of ServiceStack.Text which is a third-party library. My guess is you have a race condition in the build so it compiles against different version accidentally.

[bmwiedemann]

In strace of the build I see that no csc is called. Only /usr/lib/mono/4.5/mcs.exe and /usr/lib/mono/xbuild/14.0/bin/xbuild.exe and neither has a -deterministic option documented.

I also try to reduce races by compiling in 1-core-VMs but that seems insufficient here.

[vargaz]

The mono build is currently not deterministic:

• mcs is probably not deterministic it needs a --deterministic flag which computes the assembly guid based on the file contents, same as csc.
• the runtime executable embeds a timestamp, see mono/mini/buildver-sgen.h.

[bmwiedemann]

Here is another testcase in code that might be more familiar:

Comparing mono-addins-1.3.3-0.0.noarch.rpm to mono-addins-1.3.3-0.0.noarch.rpm
comparing the rpm tags of mono-addins
--- old-rpm-tags
+++ new-rpm-tags
@@ -39,7 +39,7 @@
 mono(Mono.Addins.CecilReflector) 32776 1.0.0.0
 mono(Mono.Addins.Gui) 32776 1.0.0.0
 mono(Mono.Addins.Setup) 32776 1.0.0.0
-mono(mautil) 32776 1.0.7527.5251
+mono(mautil) 32776 1.0.7527.5274

--- old /usr/lib/mono/mono-addins/mautil.exe (monodis)
+++ new /usr/lib/mono/mono-addins/mautil.exe (monodis)
@@ -48,7 +48,7 @@
                61 79 4E 61 6D 65 00                            ) // ayName.

   .hash algorithm 0x00008004
-  .ver  1:0:7527:5251
+  .ver  1:0:7527:5274
 }
 .module mautil.exe // GUID = { 42 }

and the difference in the last two version numbers is larger when the time difference between builds is larger. Thanks @vargaz for the pointer to buildver-sgen.h
At least https://github.com/mono/mono/blob/4835e2947/mono/mini/CMakeLists.txt#L767 is already deterministic, because cmake uses SOURCE_DATE_EPOCH for its TIMESTAMP function. But the automake version still needs love.

[bmwiedemann]

mcs needs a --deterministic flag

mcs does not have a --deterministic flag - did you mean, someone needs to add it to the mcs code?

buildver-sgen.h non-determinism is fixed in master.

[vargaz]

Yes, right now it generates a random GUID and embeds it into the generated assembly. It needs to compute the GUID from the assembly contents instead. Alternatively, the build should use csc which does have a deterministic flag.