luajit has build-CPU-dependent compilation output

[bmwiedemann]

While working on reproducible builds for openSUSE, I found that
our bcc package contained binaries produced by luajit that varied depending on the build machine's CPU features.

See https://reproducible-builds.org/ for why this matters.

I traced this down to be triggered by the JIT_F_SSE4_2 flag that changes lj_str_hash.c to use lj_str_hash_opt as alternate hash function.
In other places, the easiest way to guarantee reproducible output is to sort iterations over hash keys by key value.

See also on this topic: https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/hash

[siddhesh]

I'm curious to know how you'd get reproducible output with luajit at all; its output can vary not only based on the script it compiles but also based on the inputs the script processes. I reckon string hashing ought to be the least of your concerns there.

[bmwiedemann]

based on the inputs the script processes

reproducible builds means it is possible to get the same output from the same inputs (aka determinism) anytime anywhere.

When you run luajit compilation, it already produces the same .o files from the same .lua files on different machines, as long as all of them have AVX4.2 or all of them dont have it. At least for bcc.lua

[siddhesh]

I don't think I've understood the bcc use case. Could you leave detailed instructions for me to understand and evaluate the issue over the weekend? Thanks.

[bmwiedemann]

I couldnt attach it here for some reason, so I uploaded it to
https://www.zq1.de/~bernhard/temp/bcc.lua.txt

This is somehow produced from
https://github.com/iovisor/bcc/tree/master/src/lua/bcc

I expect that to just be a normal .lua file that can be compiled.

Here is the diff from compiling on different CPUs (using qemu-kvm -cpu kvm64 for non-AVX4.2)
http://rb.zq1.de/compare.factory-20200402/bcc-compare.out

[bmwiedemann]

I found another victim of this issue:
When building openSUSE's neovim package, it calls

/usr/bin/luajit /home/abuild/rpmbuild/BUILD/neovim-0.4.3/scripts/genvimvim.lua /home/abuild/rpmbuild/BUILD/neovim-0.4.3/src/nvim /home/abuild/rpmbuild/BUILD/neovim-0.4.3/build/runtime/syntax/vim/generated.vim /home/abuild/rpmbuild/BUILD/neovim-0.4.3/build/funcs_data.mpack

And that produces CPU-type-related order variations in generated.vim
There are also such order variations in generated binaries.

[siddhesh]

Would a flag to disable this be helpful, perhaps in the build so that you have a way to always build moonjit that gives reproducible results for such cases?

We should probably look at whether the whole string internment into a hash table is worth it too; I haven't looked too closely at it TBH, just assumed that it's useful and improved upon a patch from openresty/luajit2.

[bmwiedemann]

In other places we use existence of the SOURCE_DATE_EPOCH environment variable as a flag that a reproducible build is wanted.

https://reproducible-builds.org/docs/source-date-epoch/ documents its contents and use.

If you got patches for moonjit, I can test them.

[siddhesh]

I've pushed a new flag LUAJIT_ENABLE_REPRODUCIBLE_BUILDS that should disable string hash microarchitecture autodetection. If any other customisations are needed for reproducible builds, they should go under this flag.

Sorry about the delay, but I hope this works for you. I intended to add a PR first so that I can get your feedback, but out of habit I did a git push to master :/

[bmwiedemann]

Some extra documentation on how to enable it would be good. I think it is

make LUAJIT_ENABLE_REPRODUCIBLE_BUILDS=1 ...

The comment in src/Makefile is misleading

# Reproducible builds.  Enable this option if you need output to be the same
# across CPUs.
#XCFLAGS += -DLUAJIT_ENABLE_REPRODUCIBLE_BUILDS

because that would provide it as a define to the compiler where it is not used.

I just finished test-building bcc and neovim and both come out reproducible with this moonjit patch and make LUAJIT_ENABLE_REPRODUCIBLE_BUILDS=1 ...

[siddhesh]

Fixed comment, thanks!

[bmwiedemann]

Again no PR? IMHO it should be

-#LUAJIT_ENABLE_REPRODUCIBLE_BUILDS=0
+#LUAJIT_ENABLE_REPRODUCIBLE_BUILDS=1

[siddhesh]

I just pushed it in since it was a comment fixup. Here's a PR to fix it up the way you ought to like it. Let me know if that looks good to you :)