Distribute cg_clif as rustup component on the nightly channel

[bjorn3]

This makes it possible to use cg_clif using:

$ rustup component add rustc-codegen-cranelift-preview --toolchain nightly
$ RUSTFLAGS="-Zcodegen-backend=cranelift" cargo +nightly build

cc rust-lang/compiler-team#405.
r? @Mark-Simulacrum

[bjorn3]

./x.py build seems to build another rustc stage now. Why does that happen?

[Mark-Simulacrum]

For the bootstrap tests, I'd recommend an x.py build locally that tries to reproduce it (--dry-run should help with getting quick results), and running with -vvv to see what is invoking the newly added rustc builds. I suspect it's something in the codegenbackend step in compile.rs, but haven't investigated myself (and would prefer not to).

[bjorn3]

--dry-run should help with getting quick results

Thanks for the tip!

---

I got this with download-ci-llvm:

$ ./x.py build --dry-run -v
[...]
running: /home/bjorn/Documenten/rust/build/bootstrap/debug/bootstrap build --dry-run -v
thread 'main' panicked at 'std::fs::read_to_string(ci_llvm.join("link-type.txt")) failed with No such file or directory (os error 2)', src/bootstrap/config.rs:823:33
stack backtrace:
   0: rust_begin_unwind
             at /rustc/05b6023675d77979637b04a350c85903fbf59257/library/std/src/panicking.rs:493:5
   1: std::panicking::begin_panic_fmt
             at /rustc/05b6023675d77979637b04a350c85903fbf59257/library/std/src/panicking.rs:435:5
   2: bootstrap::config::Config::parse
             at ./src/bootstrap/config.rs:823:33
   3: bootstrap::main
             at ./src/bootstrap/bin/main.rs:14:18

Disabling it fixes the problem.

[bjorn3]

Previously it would build assemble stage 1 -> (rustc stage 0 and codegen backend stage 0) and std stage 1. Now it also builds codegen backend stage 1 -> rustc stage 1. Removing the DEFAULT override in the Step implementation fixed it.

[bjorn3]

Let's check if the rustup component is correctly generated.

@bors try

[bors]

⌛ Trying commit 909dbc1857b49279156b64061c13ab55c9f03b80 with merge 4bbf0fa74cbc11268c4f6f9d641e12b360065fc1...

[bors]

☀️ Try build successful - checks-actions
Build commit: 4bbf0fa74cbc11268c4f6f9d641e12b360065fc1 (4bbf0fa74cbc11268c4f6f9d641e12b360065fc1)

[bjorn3]

I must have made a mistake somewhere. rustup-toolchain-install-master 4bbf0fa74cbc11268c4f6f9d641e12b360065fc1 -c rustc-codegen-cranelift fails, while -c miri and -c rust-analyzer work.

[bjorn3]

I think I should wire up compiler/rustc_codegen_cranelift to the tidy license checking. It isn't currently checked due to being in a different workspace as far as I know.

[bjorn3]

@bors try

[bors]

⌛ Trying commit de705b10023b1ed5b9ad92810e3a0ea1dc29caee with merge 874d9737dc9cca284c21e5b4aaf40bf57502947f...

[bjorn3]

I found two issues:

1. The CODEGEN_BACKENDS env var doesn't get passed from the CI config to src/ci/run.sh, so we don't actually ship cg_clif.
2. For some reason despite the above we still create a rustc-codegen-cranelift-preview rustup component, but it is missing the codegen backend.

Currently trying to find the cause of both issues.

Edit: #117328 seems to be a fix for both issues.

[rust-timer]

Finished benchmarking commit (6a66ca2): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.2%	[0.2%, 0.2%]	1
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 636.535s -> 637.097s (0.09%)
Artifact size: 304.40 MiB -> 304.41 MiB (0.00%)

[tgross35]

I can't be the only one with this question - how does one actually put this to use after rustup component add rustc-codegen-cranelift-x86_64-unknown-linux-gnu?

Edit: I thought #117328 landed by rustc 1.75.0-nightly (608e9682f 2023-10-29) but I guess not. Once it does, seems it will just work by passing rustc -Zcodegen-backend=??

[Skgland]

I can't be the only one with this question - how does one actually put this to use after rustup component add rustc-codegen-cranelift-x86_64-unknown-linux-gnu?

That looks like you are mixing the cranelift codegen backend component with the compilation target toolchain.

Edit: I thought #117328 landed by rustc 1.75.0-nightly (608e9682f 2023-10-29) but I guess not. Once it does, seems it will just work by passing rustc -Zcodegen-backend=??

As github sais this was merged two days ago I would have expected this to be in the current nightly and based on the PR description I would expect the following to work

1. depending on whether nightly is installed
   a. rustup toolchain install nightly
   b. rustup toolchain update
2. rustup component add rustc-codegen-cranelift-preview --toolchain nightly
3. RUSTFLAGS="-Zcodegen-backend=cranelift" cargo +nightly build

1b and 2 succeeded the later suprisingly quickly, but 3 failed with

$ RUSTFLAGS="-Zcodegen-backend=cranelift" cargo +nightly build
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/home/bennet/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc - --crate-name ___ --print=file-names -Zcodegen-backend=cranelift --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=split-debuginfo --print=crate-name --print=cfg` (exit status: 1)
  --- stderr
  error: failed to find a `codegen-backends` folder in the sysroot candidates:
         * /home/bennet/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu
         * /home/bennet/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu

[Kobzol]

The build of the cranelift component wasn't working in this PR. It was fixed in #117328, so it should hopefully work in tomorrow's nightly.

[tgross35]

I can't be the only one with this question - how does one actually put this to use after rustup component add rustc-codegen-cranelift-x86_64-unknown-linux-gnu?

That looks like you are mixing the cranelift codegen backend component with the compilation target toolchain.

I think rustc-codegen-cranelift-preview is just a shortcut to the component rustc-codegen-cranelift-x86_64-unknown-linux-gnu (host target as applicable), I just didn't know about it since it doesn't show up in rustup component list

Edit: I thought #117328 landed by rustc 1.75.0-nightly (608e9682f 2023-10-29) but I guess not. Once it does, seems it will just work by passing rustc -Zcodegen-backend=??

As github sais this was merged two days ago I would have expected this to be in the current nightly and based on the PR description I would expect the following to work

Opened two days ago but merged 15 hours ago

[g2p]

~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/manifest-rustc-codegen-cranelift-preview-x86_64-unknown-linux-gnu is just a handful of documentation files, until the pipeline is entirely working.

[tgross35]

2023-10-30 nightly indeed works! rustc -Zcodegen-backend=cranelift / RUSTFLAGS=-Zcodegen-backend=cranelift cargo build

[lqd]

@tgross35 https://bjorn3.github.io/2023/10/31/progress-report-oct-2023.html also describes the couple of other ways you can use cargo's support for selecting the backend to use.