rust libs are not reproducible from 1.75

[sundeep-kokkonda]

Hello,

We're building rust in Yocto framework for a custom target, and there we are facing a reproducible issue with the libs generated during the build. When the rust is build in two different paths the generated libs are not identical, those are getting differed in .rustc section. This is observed from the rust 1.75.0 onwards.
(We've enabled remap-debuginfo option but still this issue is happening, this option was learned from issue discussion https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639 and this was working fine till 1.74.1)

I've analyzed the changes from 1.74.1 to 1.75.0, there are changes related to Cargo, llvm & some changes w.r.t to bolt feature/optimization are there. I've tried a few builds by reverting these changes but still the issue is reproducing.

There are a few commits in 85c0ce2 for src/bootstrap/src/core/builder.rs & src/bootstrap/src/bin/rustc.rs files which have changes related to remap-debuginfo option and I suspect this commit might causing the issue. I tried build by reverting the commit & with the latest changes in 4b7e0a0 but still issue is reproduing. There is a new RUSTC_CARGO_REGISTRY_SRC_TO_REMAP is added in this commit. I added a few debug statements in here to see the build paths passed, I can see those are passed correctly to RUSTC_CARGO_REGISTRY_SRC_TO_REMAP.

Any hints on what might cause reproducible issues with this commit on rust 1.75? Let me know if any other info is needed.

Non-reproducible libs are here:

[bjorn3]

Could you see if you can extract both versions of the .rustc section, follow the instructions in https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/10 to decompress and then show a diff of the decompressed data?

Edit: Forgot that I removed metadata compression for dylibs a while ago. So you can just diff the .rustc section directly.

[saethlin]

I'm assigning myself because I'm going to try to build a tool to help with this. It might be pretty cooked.

[sundeep-kokkonda]

Could you see if you can extract both versions of the .rustc section, follow the instructions in https://internals.rust-lang.org/t/librustc-driver-so-not-reproducible/19639/10 to decompress and then show a diff of the decompressed data?

Edit: Forgot that I removed metadata compression for dylibs a while ago. So you can just diff the .rustc section directly.

Attached librustc_driver-cbccc190731a002e & libcstr-13225ad2e5d1b622 libs .rustc section diff here (There are many more libs differed).

Diff for .rustc section of librustc_driver-cbccc190731a002e (in librustc_driver the diff is huge, below image is part of it. The attached librustc_driver_report.pdf is full diff report)

librustc_driver_report.pdf

Diff for .rustc section of libcstr-13225ad2e5d1b622

[sundeep-kokkonda]

I'm assigning myself because I'm going to try to build a tool to help with this. It might be pretty cooked.

@saethlin Tool for what?

[saethlin]

Figuring out what these bytes are.

[bjorn3]

Looks like some paths aren't getting remapped as they should.

[sundeep-kokkonda]

Looks like some paths aren't getting remapped as they should.

Is there a way to pull out that info, which paths, build stage, crate info etc? The verbose build log shows me the --remap-path-prefix option is passed during build of each crate and the debug statements (I added) in 4b7e0a0 also shows correct paths.

[sundeep-kokkonda]

Edit: Forgot that I removed metadata compression for dylibs a while ago. So you can just diff the .rustc section directly.

Does this mean the -Zls options mentioned here are not supporting now?

[bjorn3]

-Zls is still supported. It doesn't show everything inside the crate metadata though. It wouldn't have caught the fact that there are unmapped paths in the crate metadata for example. Could you see if you can get the full path that differs? Your diffing program cuts it off before the end of the path.

[sundeep-kokkonda]

Could you see if you can get the full path that differs? Your diffing program cuts it off before the end of the path.

I attached the uncut report of the diff tool. I tried to read the path names with objdump/readelf/strings tools but those are in compressed format and couldn't get the proper names.

Diff for .rustc section of librustc_driver-cbccc190731a002e

As we see the below libcstr-13225ad2e5d1b622 diff, there the differed path names are clearly visible. I've re-checked the crate build command and there the '--remap-path-prefix' is passed with correct path names.

Is there a bug with these --remap-path-prefix / remap--debuginfo options in 1.75?

[bjorn3]

Looks like the vendored crate sources don't get remapped. I found #119582 from 3 weeks ago, which fixes this. It will be in rust 1.77.

[sundeep-kokkonda]

I tried build by reverting the commit & with the latest changes in 4b7e0a0 but still issue is reproduing. There is a new RUSTC_CARGO_REGISTRY_SRC_TO_REMAP is added in this commit. I added a few debug statements in here to see the build paths passed, I can see those are passed correctly to RUSTC_CARGO_REGISTRY_SRC_TO_REMAP.

@bjorn3 As I said in issue description, I tried build with that commit but still the issue is not solved.

[bjorn3]

Missed that. For libcstr does it reduce the diff by removing the path changes and just not the changes in random bytes or does it not help at all? If the former, there are probably two different issues with one being fixed by that PR.

[sundeep-kokkonda]

For libcstr does it reduce the diff by removing the path changes and just not the changes in random bytes or does it not help at all?

In our build env, we are using vendored sources of rust and so we must take the path changes given in 4b7e0a0.
Without that patch the build crashes as there mentioned a known bug // FIXME: handle vendored sources in original sources. So, in our case there is no question about with & without path changes.

Also, I tried build with 1.76.0 release also and the issue reproduces there as well.

[sundeep-kokkonda]

I tried reproducing this issue with 1.76.0 upstream rust sources (w/o Yocto build framework) and it's reproducing there as well. The patch 4b7e0a0 also not fixing the issue.

Add below code in shell script and execute that will reproduce the issue. (Here also, the --remap-path-prefix is passed & remap-debuginfo is enabled.)

#!/bin/sh

cwd=`pwd`
echo $cwd

wget https://static.rust-lang.org/dist/rustc-1.76.0-src.tar.xz

for i in {1..2}; do
    mkdir rr$i
    cd rr$i
    echo "RUSTFLAGS=\"--remap-path-prefix `pwd`/=/home/rust/\""
    tar -xf ../rustc-1.76.0-src.tar.xz
    cd rustc-1.76.0-src
    cp config.example.toml config.toml
    sed -i '293i\vendor = true' ./config.toml
    sed -i '294 d' ./config.toml
    sed -i '681i\remap-debuginfo = true' ./config.toml
    sed -i '682 d' ./config.toml
    
    #PATCH :: hardcodepaths (This piece of code adds hardcoded paths to binary, shall be removed for reproducible binaries)
    sed -i '149,187 d' ./compiler/rustc_codegen_llvm/src/context.rs

    #PATCH :: Handle vendored sources when remapping paths
    #https://github.com/rust-lang/rust/commit/4b7e0a0b56aa2446e670dfd6558380a1039a86aa
    sed -i '1804,1813 d' ./src/bootstrap/src/core/builder.rs
    sed -i '1804i\
            let mut env_var = OsString::new();\
            if self.config.vendor {\
                let vendor = self.build.src.join("vendor");\
                env_var.push(vendor);\
                env_var.push("=/rust/deps");\
            } else {\
                let registry_src = t!(home::cargo_home()).join("registry").join("src");\
                for entry in t!(std::fs::read_dir(registry_src)) {\
                    if !env_var.is_empty() {\
                        env_var.push("\t");\
                    }\
                    env_var.push(t!(entry).path());\
                    env_var.push("=/rust/deps");\
            }\
        }' ./src/bootstrap/src/core/builder.rs

    RUSTFLAGS="--remap-path-prefix `pwd`/=/home/rust/" ./x.py -v build --stage 2
    cd $cwd
done

diff rr1/rustc-1.76.0-src/build/x86_64-unknown-linux-gnu/stage2/lib/librustc_driver-04c35085dd8f9725.so rr2/rustc-1.76.0-src/build/x86_64-unknown-linux-gnu/stage2/lib/librustc_driver-04c35085dd8f9725.so

echo "DONE"

Below are the *.so files differed with rust upstream sources and the same with Yocto build also.

[sundeep-kokkonda]

I've further analyzed the differences in 1.74 to 1.75 generated libs and intermediate files like .rlib.

During the build absolute path names are found in many stages but all are replaced/remapped as per '--remap-path-prefix' command by end of the build except the '.rustc' section contents.

I see that the .rustc section of 1.74 libs having the path names with '/rust/$hash' format. Below is the contents of libdarling_macro-*.so file.
/rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/vendor/darling_macro/src/lib.rs

Where as in 1.75 this is having absolute path names
/home/skokkond/rust/rustc-1.75.0-src/vendor/darling_macro/src/lib.rs

I tried identifying the changes in 1.75 related to paths and found below. I reverted these commits but the issue is still there.

https://github.com/rust-lang/cargo/pull/12900/files
https://github.com/rust-lang/cargo/pull/12908/files
https://github.com/rust-lang/cargo/pull/12625/files

I find a few other commits as well (listed below). I tried reverting these, but these are having a lot of dependencies with other commits also and ended up in lot of build issues.

eccc9e6
351d532
04fa124 and so on...

In summary, with 1.75

• The .rustc section having absolute path names instead of /rust/$hash format.
• The '.rustc' contents are not effected by '--remap-path-prefix' command.

Can you let us know,

• Do we need to set any compiler option to fix this? (We've already enabled remap-debuginfo & --remap-path-prefix)
• Is it expected the .rustc section contents are intact by remap-debuginfo & --remap-path-prefix options?
• Is there a way to remap the .rustc section path names?

[bjorn3]

I think eccc9e6#diff-a83e0054300d480e1f72a82b5b49c0f01c5f4e49475b0ed2427074d14c67f82cR1262 is the problem. Try removing the early return at

rust/compiler/rustc_session/src/session.rs

Line 909 in b8de591

CrateType::ProcMacro => return false,

[sundeep-kokkonda]

I think eccc9e6#diff-a83e0054300d480e1f72a82b5b49c0f01c5f4e49475b0ed2427074d14c67f82cR1262 is the problem. Try removing the early return at

rust/compiler/rustc_session/src/session.rs

Line 909 in b8de591

CrateType::ProcMacro => return false,

This works... I changed to below.
CrateType::ProcMacro => return true,

Just verified binary comparison and that shows binaries are identical... Not yet looked into binary section details yet. I will update you... @bjorn3 Thank you..

[bjorn3]

With binaries you also mean all .so files that currently differ? If so that is great!

[sundeep-kokkonda]

With binaries you also mean all .so files that currently differ? If so that is great!

Yes, I missed to mention the shared objects.

I checked the .rustc section after this change and the path names inside the .so are changed back to /rust/$hash format.

What exactly the below change is doing?
CrateType::ProcMacro => return true,

And, can this change be included in upcoming releases?

[davidtwco]

This has a fix in #121959 but it still needs a test.

[workingjubilee]

It looks like the shell script in this comment #120825 (comment) should be converted to a test. Possibly an rmake test.