Rust building and testing fails while bootstrapping it in poky

[Yashinde145]

Getting error while building & testing rust 1.72 via the src/bootstrap/bootstrap.py file in poky (cmd- bitbake rust and then python3 src/bootstrap/bootstrap.py test test_suites_names --target x86_64-poky-linux-gnu)in poky environment. Until rust 1.70 I was able to build & test the rust sources successfully.
But after updating to rust 1.72, there's an assertion failure causing the problem. The rust build fails as follows-

Note that not all variants of mir-opt tests are going to be blessed, as no mapping between a 
32bit and a 64bit target was found for x86_64-poky-linux-gnu.
You can add that mapping by changing MIR_OPT_BLESS_TARGET_MAPPING in src/bootstrap/test.rs
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `x86_64-unknown-linux-gnu`,
 right: `x86_64-poky-linux-gnu`: Cannot obtain compiler for non-native build triple at stage 0', compile.rs:1474:13

The code at compile.rs:1474

1470     fn run(self, builder: &Builder<'_>) -> Compiler {
1471         let target_compiler = self.target_compiler;
1472 
1473         if target_compiler.stage == 0 {
1474             assert_eq!(
1475                 builder.config.build, target_compiler.host,
1476                 "Cannot obtain compiler for non-native build triple at stage 0"
1477             );
1478             // The stage 0 compiler for the build triple is always pre-built.
1479             return target_compiler;
1480         }

Build terminates as follows-

test [mir-opt] tests/mir-opt/reference_prop.rs ... ok
test [mir-opt] tests/mir-opt/inline/polymorphic_recursion.rs ... ok
test result: ok. 249 passed; 0 failed; 13 ignored; 0 measured; 0 filtered out; finished in 1.39s

error: the option `Z` is only accepted on the nightly compiler
note: selecting a toolchain with `+toolchain` arguments require a rustup proxy; see <https://rust-lang.github.io/rustup/concepts/index.html>
help: consider switching to a nightly toolchain: `rustup default nightly`
note: for more information about Rust's stability policy, see <https://doc.rust-lang.org/book/appendix-07-nightly-rust.html#unstable-features>
thread 'main' panicked at 'failed to gather the target spec for i686-unknown-linux-gnu', synthetic_targets.rs:66:9
Build completed unsuccessfully in 0:05:27

Version it worked on

It most recently worked on Rust 1.70 and earlier versions

Version with regression

Rust 1.72

Backtrace

Backtrace

stack backtrace:                                                                                                                                                                                                
   0: rust_begin_unwind                                                                                                                                                                                         
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/std/src/panicking.rs:593:5                                                                                                              
   1: core::panicking::panic_fmt                                                                                                                                                                                
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:67:14                                                                                                             
   2: core::panicking::assert_failed_inner                                                                                                                                                                      
   3: core::panicking::assert_failed                                                                                                                                                                            
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/panicking.rs:228:5                                                                                                             
   4: <bootstrap::compile::Assemble as bootstrap::builder::Step>::run                                                                                                                                           
             at ./src/bootstrap/compile.rs:1474:13                                                                                                                                                              
   5: bootstrap::builder::Builder::ensure                                                                                                                                                                       
             at ./src/bootstrap/builder.rs:2055:23                                                                                                                                                              
   6: bootstrap::builder::Builder::compiler                                                                                                                                                                     
             at ./src/bootstrap/builder.rs:943:9                                                                                                                                                                
   7: <bootstrap::test::Compiletest as bootstrap::builder::Step>::run                                                                                                                                           
             at ./src/bootstrap/test.rs:1462:24                                                                                                                                                                 
   8: bootstrap::builder::Builder::ensure                                                                                                                                                                       
             at ./src/bootstrap/builder.rs:2055:23                                                                                                                                                              
   9: <bootstrap::test::UiFullDeps as bootstrap::builder::Step>::run                                                                                                                                            
             at ./src/bootstrap/test.rs:1260:17                                                                                                                                                                 
  10: bootstrap::builder::Builder::ensure                                                                                                                                                                       
             at ./src/bootstrap/builder.rs:2055:23                                                                                                                                                              
  11: <bootstrap::test::UiFullDeps as bootstrap::builder::Step>::make_run                                                                                                                                       
             at ./src/bootstrap/test.rs:1256:17                                                                                                                                                                 
  12: bootstrap::builder::StepDescription::maybe_run                                                                                                                                                            
             at ./src/bootstrap/builder.rs:278:13                                                                                                                                                               
  13: bootstrap::builder::StepDescription::run                                                                                                                                                                  
             at ./src/bootstrap/builder.rs:315:21                                                                                                                                                               
  14: bootstrap::builder::Builder::run_step_descriptions                                                                                                                                                        
             at ./src/bootstrap/builder.rs:935:9                                                                                                                                                                
  15: bootstrap::builder::Builder::execute_cli                                                                                                                                                                  
             at ./src/bootstrap/builder.rs:916:9                                                                                                                                                                
  16: bootstrap::Build::build                                                                                                                                                                                   
             at ./src/bootstrap/lib.rs:697:17                                                                                                                                                                   
  17: bootstrap::main                                                                                                                                                                                           
             at ./src/bootstrap/bin/main.rs:74:5                                                                                                                                                                
  18: core::ops::function::FnOnce::call_once                                                                                                                                                                    
             at /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/ops/function.rs:250:5                                                                                                          
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.                                                                                                                         
Build completed unsuccessfully in 0:00:20


<!--
@rustbot modify labels: +regression-from-stable-to-{stable} -regression-untriaged 

[Nilstrieb]

I believe MIR_OPT_BLESS_TARGET_MAPPING is unrelated to the issue you're seeing, but I'm not sure.

[Yashinde145]

Yeah, I guess the assertion failure is main cause.
Why the two strings are different though? Until rust 1.70, I didn't faced this assertion failure(diff in builder.config.build i.e left: x86_64-unknown-linux-gnu, and target_compiler.host i.e right: x86_64-poky-linux-gnu). When did git blame for these lines, found that this code was present (from 5 years i.e 2018)even in the earlier rust versions.

[apiraino]

Adding useful labels for triaging

@rustbot label -I-prioritize +T-bootstrap -needs-triage

[onur-ozkan]

I couldn't reproduce this problem from the e39976f HEAD.

The assertation passes as expected(on `./x test bootstrap --target=..), I also printed them out.

The diff:

Sorry for sharing the the outputs in the image format; the attempt was done inside of QEMU(Debian 11).

[Yashinde145]

Thanks for checking @onur-ozkan,
But, my build shows following-
Note- The targets are printed 3 times and the assertion fails in the 3rd call

Compiling clap v4.2.4
   Compiling clap_complete v4.2.2
    Finished dev [unoptimized] target(s) in 20.26s

builder.config.build = x86_64-unknown-linux-gnu
target_compiler.host = x86_64-unknown-linux-gnu
builder.config.build = x86_64-unknown-linux-gnu
target_compiler.host = x86_64-unknown-linux-gnu
builder.config.build = x86_64-unknown-linux-gnu
target_compiler.host = x86_64-poky-linux-gnu

Moreover, if I bypass the assertion check and then build, why do the compiler asks to switch to nightly channel instead of continuing to stable? I am wondering how does it gets the Z option during the build since I did not not provide any such options.

test [mir-opt] tests/mir-opt/reference_prop.rs ... ok
test [mir-opt] tests/mir-opt/inline/polymorphic_recursion.rs ... ok
test result: ok. 249 passed; 0 failed; 13 ignored; 0 measured; 0 filtered out; finished in 1.39s

error: the option `Z` is only accepted on the nightly compiler
note: selecting a toolchain with `+toolchain` arguments require a rustup proxy; see <https://rust-lang.github.io/rustup/concepts/index.html>
help: consider switching to a nightly toolchain: `rustup default nightly`
note: for more information about Rust's stability policy, see <https://doc.rust-lang.org/book/appendix-07-nightly-rust.html#unstable-features>
thread 'main' panicked at 'failed to gather the target spec for i686-unknown-linux-gnu', synthetic_targets.rs:66:9
Build completed unsuccessfully in 0:05:27

Let me know if I am missing something

[onur-ozkan]

Do you have a custom config.toml?

[Yashinde145]

The config.toml generated during the rust build is as follows-

build_dir is the directory where I am building rust (using cmd bitbake rust in yocto poky sources)

[target.x86_64-poky-linux-gnu]                                                                                                                                                                                  
llvm-config = "build_dir/recipe-sysroot/usr/lib/llvm-rust/bin/llvm-config"                                                                         
cxx = "build_dir/wrapper/target-rust-cxx"                                                                                                          
cc = "build_dir/wrapper/target-rust-cc"                                                                                                            
linker = "build_dir/wrapper/target-rust-ccld"                                                                                                      
                                                                                                                                                                                                                
[target.x86_64-unknown-linux-gnu]                                                                                                                                                                               
llvm-config = "build_dir/recipe-sysroot-native/usr/lib/llvm-rust/bin/llvm-config"                                                                  
cxx = "build_dir/wrapper/build-rust-cxx"                                                                                                           
cc = "build_dir/wrapper/build-rust-cc"                                                                                                             
linker = "build_dir/wrapper/build-rust-ccld"                                                                                                       
                                                                                                                                                                                                                
[llvm]                                                                                                                                                                                                          
static-libstdcpp = false                                                                                                                                                                                        
                                                                                                                                                                                                                
[rust]                                                                                                                                                                                                          
rpath = true                                                                                                                                                                                                    
channel = "stable"                                                                                                                                                                                              
optimize = true                                                                                                                                                                                                 
verbose-tests = true                                                                                                                                                                                            
                                                                                                                                                                                                                
[build]                                                                                                                                                                                                         
submodules = false                                                                                                                                                                                              
docs = false                                                                                                                                                                                                    
rustc = "build_dir/rust-snapshot/bin/rustc"                                                                                                        
profiler = true                                                                                                                                                                                                 
cargo = "build_dir/rust-snapshot/bin/cargo"                                                                                                        
vendor = true                                                                                                                                                                                                   
target = ["x86_64-poky-linux-gnu"]                                                                                                                                                                              
host = ["x86_64-poky-linux-gnu"]                                                                                                                                                                                
build = "x86_64-unknown-linux-gnu"                                                                                                                                                                              
                                                                                                                                                                                                                
[install]                                                                                                                                                                                                       
prefix = "build_dir/image/usr"                                                                                                                     
bindir = "build_dir/image/usr/bin"                                                                                                                 
libdir = "build_dir/image/usr/lib"                                                                                                                 
datadir = "build_dir/image/usr/share"                                                                                                              
mandir = "build_dir/image/usr/share/man"

[onur-ozkan]

iirc bitbake rust will give you the stable 1.68 compiler. If so, please remove build.cargo and build.rust from your configuration, and use the beta compiler which will be downloaded from CI automatically during the bootstrapping.

[Yashinde145]

Actually, bitbake rust will give stable 1.70 compiler(on master branch). I am manually upgrading the version to 1.72 and then getting the error in bootstrapping build.

I didn't understand removing build.cargo and build.rust and using beta compiler from CI during bootstrapping. Can you enlighten me on it?

[onur-ozkan]

Actually, bitbake rust will give stable 1.70 compiler(on master branch).

Still, this is wrong.

I didn't understand removing build.cargo and build.rust and using beta compiler from CI during bootstrapping. Can you enlighten me on it?

Remove the following lines:

rustc = "build_dir/rust-snapshot/bin/rustc"                                                                                                        
cargo = "build_dir/rust-snapshot/bin/cargo"                                                                                                        

from your configuration

[Yashinde145]

Ok, will check and get back.

[Yashinde145]

@onur-ozkan Thanks for the suggestion, but I can't modify config.toml. The same config.toml worked with previous rust versions. Actually, the build compiler fetches some -Z (nightly build flag option) while building on the stable channel and it's recommended not to shift to nightly build. Is there any way I can find which flag causes this build failure and suppress that flag to proceed the build?

[Yashinde145]

@onur-ozkan, any updates?

[onur-ozkan]

Sorry, I somehow missed the notification.

Bootstrap requires the immediate previous version to compile the next version, allowing it to handle unstable features. So there's an issue with your configuration (rustc and cargo), and you should find a way to update your config file. I'm surprised you didn't encounter an error message like: Unexpected rustc version: 1.70.0; we should use **** to build the source with ****. It might be more helpful to receive this error message at the beginning of the bootstrap process, I suppose.

[Yashinde145]

@onur-ozkan

Actually, I see that -Zdoctest-in-workspace a nightly flag is passed by default in Rust 1.72 release notes here

More info- docs rust-lang/cargo#12221 rust-lang/cargo#12288

I suspect that -Zdoctest-in-workspace is causing buid failure as seen in above comment here, #115642 (comment).

[onur-ozkan]

As I explained above, you cannot build upstream rustc with 1.70 anymore, even for non-poky environments. You should either not set rustc manually in the build configuration or set it to the proper version.

[onur-ozkan]

#119619 PR will close this issue upon merging, as we have a separate issue (#97669) open for the other problem.

[Yashinde145]

@onur-ozkan , Thanks for the help to fix this issue.

Here we have an issue for the remote-test tool problem #97669
It seems to occur randomly in QEMU environments from time to time. Retrying seems to fix the issue for each occurrence. I will try to identify the root cause in the coming week.

Yeah, I faced this panic failures randomly after the test suite testing is completed starting from rust 1.71 version

[Yashinde145]

Just remembered now,

How do I handle the assertion mentioned in #115642 (comment)?
Until now I was by-passing the assertion check with the following changes:
Should I continue with it?

diff --git a/src/bootstrap/src/core/build_steps/compile.rs b/src/bootstrap/src/core/build_steps/compile.rs                          
--- a/src/bootstrap/src/core/build_steps/compile.rs                                                                                 
+++ b/src/bootstrap/src/core/build_steps/compile.rs                                                                                 
@@ -1533,13 +1533,15 @@                                                                                                              
     /// must have been previously produced by the `stage - 1` builder.build                                                        
     /// compiler.                                                                                                                  
     fn run(self, builder: &Builder<'_>) -> Compiler {                                                                              
-        let target_compiler = self.target_compiler;                                                                                
+        let mut target_compiler = self.target_compiler;                                                                            
+        target_compiler.host = builder.config.build;                                                                                                                                              
                                                                                                                                    
         if target_compiler.stage == 0 {                                                                                            
             assert_eq!( 
                       builder.config.build, target_compiler.host,
                       "Cannot obtain compiler for non-native build triple at stage 0"
                       );
             // The stage 0 compiler for the build triple is always pre-built.
             return target_compiler;
         }

[onur-ozkan]

I will add another commit to handle that.

update: e7fc441

[Yashinde145]

Do we still need the patch mentioned in #115642 (comment) along with the changes in #119619 PR?

I think the three file changes in the #119619 PR should be sufficient, right?

[onur-ozkan]

Do we still need the patch mentioned in #115642 (comment) along with the changes in #119619 PR?

I think the three file changes in the #119619 PR should be sufficient, right?

#119619 PR is enough, you don't need any other patches.

[Yashinde145]

Here we have an issue for the remote-test tool problem #97669

It seems to occur randomly in QEMU environments from time to time. Retrying seems to fix the issue for each occurrence. I will try to identify the root cause in the coming week.

Hi @onur-ozkan,
Did you get a chance to look at the remote test-tool problem?.

[onur-ozkan]

Did you get a chance to look at the remote test-tool problem?

Not yet

[onur-ozkan]

Are you able to run any tests with remote-test or does it give the error instantly at the beginning?

[Yashinde145]

Yes, I am able to run the tests. The error is seen at
Building stage0 tool remote-test-server (x86_64-unknown-linux-gnu -> x86_64-poky-linux-gnu)

[onur-ozkan]

I meant sending tests and running in the qemu environment

I tested with the two commits and the errors of using nightly flag and custom target specs were not seen. Testing was completed for the test suites like ui, run-pass-valgrind, coverage, mir-opt, codegen, assembly, incremental.

But, after that while coping libs to qemu emulator(we run the test suite on qemu), following io::copy panic was observed:

 REMOTE copy libs to emulator (x86_64-poky-linux-gnu)                                                                          
 Building stage0 tool remote-test-server (x86_64-unknown-linux-gnu -> x86_64-poky-linux-gnu)
 
                        
 done pushing "/home/build-st/tmp/work/core2-64-poky-linux/rust/1.75.0/rustc-1.75.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-poky-linux-gnu/lib/libcstr-d49e684941d617ce.so"                                   
 done pushing "/home/build-st/tmp/work/core2-64-poky-linux/rust/1.75.0/rustc-1.75.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-poky-linux-gnu/lib/libzerovec_derive-52a1b6c914833752.so"                         
 done pushing "/home/build-st/tmp/work/core2-64-poky-linux/rust/1.75.0/rustc-1.75.0-src/build/x86_64-unknown-linux-gnu/stage0-sysroot/lib/rustlib/x86_64-poky-linux-gnu/lib/librustc_fluent_macro-4b4331439c3bd942.so"
 
thread 'main' panicked at src/tools/remote-test-client/src/main.rs:353:5: 
io::copy(&mut file, dst) failed with Connection reset by peer (os error 104)                                                  
 stack backtrace:                                                                                                              
    0: rust_begin_unwind                                                                                                       
              at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/std/src/panicking.rs:597:5                            
    1: core::panicking::panic_fmt                                                                                              
              at /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/panicking.rs:72:14                           
    2: remote_test_client::send                                                                                                
    3: remote_test_client::main                                                                                                
 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.                                       
 Build completed unsuccessfully in 0:09:27

Is the error always the same, like this?

[Yashinde145]

Yes, the error is always same.

[onur-ozkan]

Can you try this patch onur-ozkan@43183f7 and tell me if it fixes the problem?

[Yashinde145]

The onur-ozkan@43183f7 patch does not fix the problem. Still, the same error is seen.

[onur-ozkan]

I was able to repreduce the io::copy(&mut file, dst) failed with Connection reset by peer (os error 104) error, but only by sending large files (and the patch I linked fixes it).

Unfortunately I don't know your use case, and I can't invest time on #115642 (comment) for debugging or trying to reproduce the error. If you can't debug and fix the problem you get, I suggest you to give more context about your use case (configurations, commands, etc.) under the #97669 issue.

[Yashinde145]

I got some work around for this,
by excluding ui-fulldeps test suite and src/tools/replace-version-placeholder, src/tools/jsondoclint tests and it works for x86_64-poky-linux-gnu.

But, when I check for other cross-compiled targets(x86 32 bit, arm 32 & 64 bits and mips64), the following lib failure is seen

error: linking with `cc` failed: exit status: 1

build-st/tmp/hosttools/ld:  build-st/tmp/work/core2-32-poky-linux/rust/1.74.1/rustc-1.74.1-src/build/x86_64-unknown-linux-gnu/stage1-std/i686-poky-linux-gnu-synthetic-miropt-abort/release/deps/std-55f86614dec482ec.3j457j23godxdmvr.rcgu.o: file class ELFCLASS32 incompatible with ELFCLASS64
build-st/tmp/hosttools/ld: final link failed: file in wrong format

collect2: error: ld returned 1 exit status
error: could not compile `std` (lib) due to previous error

I am investigating the problem and will let you know if there are any updates.

[onur-ozkan]

That seems to be related with MirOptPanicAbortSyntheticTarget tests

rust/src/bootstrap/src/core/build_steps/test.rs

Lines 1530 to 1534 in f9c2421

	let panic_abort_target = builder.ensure(MirOptPanicAbortSyntheticTarget {
	compiler: self.compiler,
	base: target,
	});
	run(panic_abort_target);

[Yashinde145]

Yes, the log says the same and the error was seen after MirOptPanicAbortSyntheticTarget test cases while building stage1 library artifacts.

Meanwhile, I came across the following and have a doubt,
As mentioned in #40049 (comment), do we have to pass the -C linker= flag to RUSTFLAGS explicitly for the linker error in #115642 (comment)?

Earlier it was working without passing the flag explicitly.

[onur-ozkan]

Earlier we didn't have those tests, so they might require passing -C linker=. You can try and see if that solves the problem.

[Yashinde145]

That seems to be related with MirOptPanicAbortSyntheticTarget tests

rust/src/bootstrap/src/core/build_steps/test.rs

Lines 1530 to 1534 in f9c2421

	let panic_abort_target = builder.ensure(MirOptPanicAbortSyntheticTarget {
	compiler: self.compiler,
	base: target,
	});
	run(panic_abort_target);

Is there any way to handle this panic failure and make the tests run successfully?

[onur-ozkan]

You can add --skip mir-opt to your command to exclude mir-opt tests, then run x test mir-opt to run mir-opt tests without MirOptPanicAbortSyntheticTarget.