.beam files differ between builds

[bmwiedemann]

Environment

• Elixir & Erlang/OTP versions (elixir --version):
  elixir-1.7.4 erlang-21.1.4
• Operating system: openSUSE Tumbleweed 20190124

Current behavior

Somewhat similar to #4814 building the openSUSE elixir package produces various .beam files that differ for every build, e.g. /usr/lib/elixir/lib/mix/ebin/Elixir.Mix.Tasks.Deps.Unlock.beam

Some of the diff is even there when doing builds as similar as possible (e.g. disabling ASLR, 1-core VM without parallelism)

The erlang package itself does not have this kind of variations in its .beam files (but others from timestamps)

Note: the elixir-ex_doc package (in the best case) only has 1 beam file with 1 bit diff so might be easier to debug.

Expected behavior

Builds should produce deterministic results.
See https://reproducible-builds.org/ for why this matters.

[josevalim]

Thanks @bmwiedemann. Do you have steps to reproduce this? How are you going the builds? Which Erlang/OTP version are you using?

I am getting consistent results with Elixir master and Erlang/OTP 21. Note that earlier Erlang/OTP versions would include timestamps, which would make them not reproducible.

[josevalim]

I also could not reproduce this in ExDoc:

~/OSS/ex_doc[master]$ mix compile --force
Compiling 18 files (.ex)
Generated ex_doc app
~/OSS/ex_doc[master]$ md5 _build/dev/lib/ex_doc/ebin/*
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.CLI.beam) = 48e25a8df33d27d27f9f556db1b52a41
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Config.beam) = 28887a5beed69bcecfc76074dd7a9e3d
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.Assets.beam) = cb365ebd1781bf25bd36d6e095b5eed9
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.Templates.beam) = c7589f848178d992a962b2d488c86b0a
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.beam) = a27fa4cfe66a6c20f71e74fced081456
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Assets.beam) = 5a55eaea4239a96ed9a0b435c73c2285
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Autolink.beam) = 3ec5d6a224a44c7c63bf3666518e08f4
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam) = ffaaa7c414f24b76a1459b5ac4ae7807
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.beam) = 99b9c8836e9c529349e1f14ca9b1b655
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.FunctionNode.beam) = c5f4042c02cdd47be20cf7a518367fce
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.GroupMatcher.beam) = fb0135e08cc44e59ef05400cfd6f8a06
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Highlighter.beam) = 20f15b6cedae7a7f4a1c8fe60a39b031
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.Cmark.beam) = a0fbb555b7ce8dc580b5aa91786c4758
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.Earmark.beam) = 94b664aecbb6e9143eeac3d35369d06f
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.beam) = 28811e650fd5778f4f2e7011b1442f32
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.ModuleNode.beam) = 7422d37d42ecfddcb44b12cdcb65bc3b
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Retriever.Error.beam) = 012642520a72ed473a319f034c078e40
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Retriever.beam) = 9be29c27cc206b165680989f4d277343
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.TypeNode.beam) = 7fcfac1158bdea265885d8fbe153ca2c
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.beam) = a62fdf39d3f2c13444852e0ef8e77ea4
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.Mix.Tasks.Docs.beam) = ac31564174cabc7b26faa2b592849f38
MD5 (_build/dev/lib/ex_doc/ebin/ex_doc.app) = 64735fd5d2ccd3ef261675b9c56fe3f0
~/OSS/ex_doc[master]$ mix compile --force
Compiling 18 files (.ex)
Generated ex_doc app
~/OSS/ex_doc[master]$ md5 _build/dev/lib/ex_doc/ebin/*
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.CLI.beam) = 48e25a8df33d27d27f9f556db1b52a41
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Config.beam) = 28887a5beed69bcecfc76074dd7a9e3d
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.Assets.beam) = cb365ebd1781bf25bd36d6e095b5eed9
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.Templates.beam) = c7589f848178d992a962b2d488c86b0a
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.EPUB.beam) = a27fa4cfe66a6c20f71e74fced081456
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Assets.beam) = 5a55eaea4239a96ed9a0b435c73c2285
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Autolink.beam) = 3ec5d6a224a44c7c63bf3666518e08f4
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam) = ffaaa7c414f24b76a1459b5ac4ae7807
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.beam) = 99b9c8836e9c529349e1f14ca9b1b655
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.FunctionNode.beam) = c5f4042c02cdd47be20cf7a518367fce
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.GroupMatcher.beam) = fb0135e08cc44e59ef05400cfd6f8a06
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Highlighter.beam) = 20f15b6cedae7a7f4a1c8fe60a39b031
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.Cmark.beam) = a0fbb555b7ce8dc580b5aa91786c4758
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.Earmark.beam) = 94b664aecbb6e9143eeac3d35369d06f
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Markdown.beam) = 28811e650fd5778f4f2e7011b1442f32
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.ModuleNode.beam) = 7422d37d42ecfddcb44b12cdcb65bc3b
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Retriever.Error.beam) = 012642520a72ed473a319f034c078e40
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.Retriever.beam) = 9be29c27cc206b165680989f4d277343
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.TypeNode.beam) = 7fcfac1158bdea265885d8fbe153ca2c
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.ExDoc.beam) = a62fdf39d3f2c13444852e0ef8e77ea4
MD5 (_build/dev/lib/ex_doc/ebin/Elixir.Mix.Tasks.Docs.beam) = ac31564174cabc7b26faa2b592849f38
MD5 (_build/dev/lib/ex_doc/ebin/ex_doc.app) = 64735fd5d2ccd3ef261675b9c56fe3f0

[bmwiedemann]

erlang is 21.1.4

Here is a reproducer (also works in Debian, if you apt install osc obs-build)
needs a free openSUSE account, though.

osc checkout openSUSE:Factory/elixir-ex_doc ; cd $_
osc build --noservice
md5sum /var/tmp/build-root/standard-x86_64/home/abuild/rpmbuild/BUILD/ex_doc-v0.7.2/_build/prod/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam

produced here:
b02d8e1067bb0df91e164fac9dc46457 /var/tmp/build-root/standard-x86_64/home/abuild/rpmbuild/BUILD/ex_doc-v0.7.2/_build/prod/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam
460044305532c62996b564af01726f24 /var/tmp/build-root/standard-x86_64/home/abuild/rpmbuild/BUILD/ex_doc-v0.7.2/_build/prod/lib/ex_doc/ebin/Elixir.ExDoc.Formatter.HTML.Templates.beam

Logs show, it just uses mix compile, but our build scripts scratch the whole BUILD dir to ensure a clean build.

[josevalim]

Oh, sorry, I see now that you included Erlang in your initial report, sorry.

Now I also see why I can't reproduce it, I am running on v1.8.0 which has this commit: d76f150

The commit above made the compiler deterministic when meta-programming. Therefore this is fixed in v1.8+ and master. In any case, thank you for checking and for the report!

[bmwiedemann]

d76f150 cherry-picked cleanly to 1.7.4 and reduced the elixir diff to only have instances of

/usr/lib/elixir/lib/ex_unit/ebin/Elixir.ExUnit.Assertions.beam differs at offset '8' (Erlang BEAM file)
@@ -1,4 +1,4 @@
-00000000  46 4f 52 31 00 00 6f 74  42 45 41 4d 41 74 55 38  |FOR1..otBEAMAtU8|
+00000000  46 4f 52 31 00 00 6f 78  42 45 41 4d 41 74 55 38  |FOR1..oxBEAMAtU8|
 00000010  00 00 07 41 00 00 00 af  18 45 6c 69 78 69 72 2e  |...A.....Elixir.|

Is there another patch missing?

[josevalim]

This is a bit weird. Bytes 5 to 8 are the size of the BEAM chunk. So if the size is different, then something else should also show up in the diff?

[bmwiedemann]

indeed, our special-purpose diff tool was hiding the lower part diff.txt
This now looks pretty similar to what it was without the patch.
I'm wondering if I should spend effort on packaging 1.8 first.

[josevalim]

I see, so the difference is in the documentation chunk. I am not quite sure yet though. I can also reproduce it locally. I will take a look later after I pick the kids from school.

[josevalim]

Btw, this diff tool is neat, is it available externally? :D

[bmwiedemann]

The bad one is build-compare and the good one is https://github.com/bmwiedemann/reproducibleopensuse/blob/master/filterdiff
used as filterdiff xxd $file1 $file2
or filterdiff "hexdump -C" $file1 $file2

[bmwiedemann]

If you want to work with the openSUSE package, you can also try my rbk tool from the reproducibleopensuse repo.
I quickly upgraded to elixir-1.8.0 in https://build.opensuse.org/package/show/home:bmwiedemann:reproducible:test/elixir and it showed a similar diff to 1.7.4 .
Using 1.8 indeed fixed the indeterminism in the elixir-ex_doc build, so that was something else to what affects .beam files in elixir itself.

[josevalim]

Got it. So we literally had :rand.uniform being executed at the moment the docs were defined. Fixed now.