Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
reproducible builds: non-deterministic use of cmpq #50556
I had issues building a deterministic binary for one of my projects, this is the reprotest command I ran:
This causes a bunch of other differences since the first version is two bytes shorter.
I'm not sure if this decision is made by llvm or rustc, but it seems this happens in a non-deterministic way.
The project I'm using to test this is rather large, sadly I can't provide an isolated example. It also seems this doesn't affect every project.
Screenshot (easier to read)
I'm seeing the same cmpq diff with rust-1.26.2 on openSUSE with several of our packages:
looking closer at exa
movdqa %xmm1,%fs:0xffffffffffffff90 mov $something,%eax movq %rax,%xmm1 - cmpq $something,%fs:0xffffffffffffff80 + mov %fs:0xffffffffffffff80,%rax movdqa %xmm1,%fs:0xffffffffffffff80 + test %rax,%rax je <_ZN3std9panicking20rust_panic_with_hook17h33ee6001a18cb890E + ofs>
and again with 3exa4main
That is some pretty obvious pattern.
e.g. for exa
+ cargo build --release -j1 Compiling winapi-build v0.1.1 - Compiling num-traits v0.1.40 Compiling libc v0.2.30 - Compiling gcc v0.3.53 - Compiling rustc-serialize v0.3.24 - Compiling pkg-config v0.3.9 + Compiling num-traits v0.1.40 Compiling winapi v0.2.8 + Compiling pkg-config v0.3.9 + Compiling rustc-serialize v0.3.24 + Compiling gcc v0.3.53 Compiling matches v0.1.6 + Compiling unicode-normalization v0.1.5 Compiling unicode-width v0.1.4 Compiling nom v1.2.4 - Compiling unicode-normalization v0.1.5 - Compiling utf8-ranges v0.1.3 Compiling percent-encoding v1.0.0 + Compiling utf8-ranges v0.1.3 Compiling regex-syntax v0.3.9 Compiling byteorder v0.4.2 - Compiling log v0.3.8 Compiling bitflags v0.9.1 + Compiling log v0.3.8 Compiling getopts v0.2.14 - Compiling ansi_term v0.8.0 - Compiling lazy_static v0.2.8 + Compiling glob v0.2.11 Compiling natord v1.0.9 + Compiling ansi_term v0.8.0 Compiling scoped_threadpool v0.1.7 - Compiling glob v0.2.11 + Compiling lazy_static v0.2.8 Compiling kernel32-sys v0.2.2 - Compiling num-integer v0.1.35 - Compiling number_prefix v0.2.7 Compiling rand v0.3.16 Compiling memchr v0.1.11 Compiling locale v0.2.2 - Compiling term_size v0.3.0 Compiling users v0.5.3 + Compiling term_size v0.3.0 Compiling num_cpus v1.6.2 - Compiling cmake v0.1.25 + Compiling num-integer v0.1.35 + Compiling number_prefix v0.2.7 Compiling num-complex v0.1.40 Compiling libz-sys v1.0.16 + Compiling cmake v0.1.25 Compiling unicode-bidi v0.3.4 Compiling pad v0.1.4 Compiling term_grid v0.1.6 Compiling iso8601 v0.1.1 - Compiling num-iter v0.1.34 - Compiling num-bigint v0.1.40 Compiling aho-corasick v0.5.3 + Compiling num-bigint v0.1.40 + Compiling num-iter v0.1.34 Compiling libgit2-sys v0.6.14 Compiling idna v0.1.4 Compiling num-rational v0.1.39
but OTOH each position in the list varies only by a bit, so maybe it is using threads internally even on a 1-core VM and does something racy.
@bmwiedemann the order of the crates doesn't matter, I currently have reprotest tests pass for a different binary that also compiles its dependencies out-of-order, it seems this is not the source of the problem.
I looking at way to make rust 1.27.2 package build reproducible in GuixSD distro and I got same issue. It's constantly happen in
For build I used external pre-build llvm 6.0.1 package. Rust 1.26.2 build is reproducible with same configurations.
Sources was built with next commands:
As result I got different code for
Do you know way how I can try to debug it or if there was fixes for this behavior in newer versions of Rust?
I'm still getting such diffs in exa-0.8.0 with rust-1.30.0 in openSUSE, even when varying as little as possible:
movdqa %xmm1,%fs:0xffffffffffffff70 mov $something,%eax movq %rax,%xmm1 - mov %fs:0xffffffffffffff60,%rax + cmpq $something,%fs:0xffffffffffffff60 movdqa %xmm1,%fs:0xffffffffffffff60 - test %rax,%rax je <_ZN3std9panicking20rust_panic_with_hook17h40c40e76fede9cc3E + ofs> movq %xmm0,%rbx test %rbx,%rbx