testing cg_gcc is difficult

[workingjubilee]

cc @GuillaumeGomez @antoyo

The backend currently only supports x86-64, but some contributors primarily use aarch64 machines, but have cause to work on the codegen backends. They instead have to force cg_gcc's code through emulators, which is often an inadequate solution. It would be better if it simply supported all of our tier 1 architectures.

Meta

rustc --version --verbose:

rustc 1.79.0-nightly (ccfcd950b 2024-04-15)
binary: rustc
commit-hash: ccfcd950b333fed046275dd8d54fe736ca498aa7
commit-date: 2024-04-15
host: x86_64-unknown-linux-gnu
release: 1.79.0-nightly
LLVM version: 18.1.3

[workingjubilee]

Hopefully this is just tweaking a build flag?

[antoyo]

It supports other architectures, but we decided to only enable x86-64 because GCC requires to build a toolchain per target, so this is going to take some CI resources to build them.

Can you describe your use case a bit more?
If this is for local development, there might be a simple solution.

[antoyo]

There might be a bug to fix for aarch64 in cg_gcc, though.

[workingjubilee]

@antoyo The Rust Foundation provides both aarch64 and x86_64 virtual desktops for the purpose of working on the compiler. So if someone starts work on a contribution using an aarch64 desktop, finds out they need to modify cg_gcc, they have to either

• learn about emulators! some will regard this as fun, at least?
• switch to an x86_64 desktop! this is... easy, kinda? but can be a pain in the ass because not everything can be easily moved via git. build directories are real and now they have to throw that out and wait however long it takes to rebuild what they were working on.

It's just these little tripping hazards which cause unnecessary stumbles for contributors. They are technically possible to overcome but it can be annoying af, yanno? I think it makes sense to support both architectures since we do have both x86_64-unknown-linux-gnu and aarch64-unknown-linux-gnu at tier 1, and building gcc is relatively easy on those platforms, and it is much easier to e.g. emulate Linux than it is to emulate another architecture. And by "emulate" that probably more specifically means "run the kernel inside a hypervisor", realistically, but that version is still comparatively seamless.

[workingjubilee]

Someone primarily using an aarch64-apple-darwin machine to contribute (and we do have a few of those) and doesn't currently have access to the virtual desktops would be stuck with Rosetta2 JITting the entire compiler, probably? And then libgccjit generates x86-64 code to compile test binaries? Which then Rosetta2 JITs? And that can be fast but... that's a bit of a lot that "ability to directly use native binaries, even if using a foreign object encoding" bypasses (e.g. they could also have Asahi or Parallels on the machine).

Any recommendations for unblocking people running into this issue should probably be PRed to the dev guide! As should... probably... more general build/test/etc. instructions for both codegen backends?

[GuillaumeGomez]

One important thing to note is that currently the GCC backend is broken in the compiler (something about statics). I'm currently (although slowly) looking into it.

There are multiple issues we need to fix to make it easier for everyone to contribute on GCC backend:

• Allow to pick which backend to use for x.py commands.
• Have more pre-compiled targets for libgccjit.so (you can build it yourself, but we need to update the guide as you mentioned).
• Fix the backend with latest rustc updates.

[workingjubilee]

Oof! That sounds "exciting".

Allow to pick which backend to use for x.py commands.

Ah, something like --cg-backend=gcc, I guess?

We'd probably want to allow setting something like --cg-bootstrap=llvm ("use LLVM to build the compiler itself") and --cg-backend=gcc ("use cg_gcc to build any other binaries") individually? Do we have an issue for that open yet?

Have more pre-compiled targets for libgccjit.so (you can build it yourself, but we need to update the guide as you mentioned).

Ah, having build instructions there would be great! Or even a pointer to build instructions from the guide into a README in the repo somewhere, if they're still likely to be updated often in tandem with cg_gcc.

[GuillaumeGomez]

Oof! That sounds "exciting".

It is. 😉

Allow to pick which backend to use for x.py commands.

Ah, something like --cg-backend=gcc, I guess?

We'd probably want to allow setting something like --cg-bootstrap=llvm ("use LLVM to build the compiler itself") and --cg-backend=gcc ("use cg_gcc to build any other binaries") individually? Do we have an issue for that open yet?

Exactly. It's in my todo list once the GCC backend is fixed (no use to add a new build system feature if it cannot be tested nor used haha).

Have more pre-compiled targets for libgccjit.so (you can build it yourself, but we need to update the guide as you mentioned).

Ah, having build instructions there would be great! Or even a pointer to build instructions from the guide into a README in the repo somewhere, if they're still likely to be updated often in tandem with cg_gcc.

There are already in the rustc_codegen_gcc repository here. It's just that I don't know how well it would integrate with the current build system. In our own build system, it's very easy to use though. Again, more plumbing to be done to make everything as smooth as it should be which I did already... if you are on x86_64 target. 😅

[antoyo]

Would pre-built libgccjit be helpful?
We already have cross-compilers from x86-64 to m68k and arm in this repo.
We could have some (cross-)compilers from aarch64 if needed.

[workingjubilee]

I think pre-built libgccjit would be good, but I also think sufficient build/cross-build tools/instructions available that it's easy enough to build libgccjit on one's own should be fine? It seems reasonable to expect compiler contributors to put up with inconveniences occasionally, but a lot of the sorts of persons who will be contributing to the compiler are the sort of person who will want to do something "properly" so it won't actually occur to them to simply give up and switch to x86-64, they'll look at how to get cg_gcc working "the right way" on their system first. So even instructions to give up would be helpful. ^^;

[antoyo]

As mentionned by @GuillaumeGomez, the repo contains the build instructions.
What would you like to see?
Would you want these instructions in the dev guide instead?

[workingjubilee]

So I checked the rustc_codegen_gcc directory/repo and...

This requires a patched libgccjit in order to work. You need to use my fork of gcc which already includes these patches.

...hm. what are the instructions for building it? when I looked for them, I couldn't find them in the README of the rustc_codegen_gcc directory, so I checked the README in the gcc dir.

I skimmed and found

• The directory INSTALL contains copies of the installation information

• See the file gcc/doc/gcc.texi (together with other files that it
  includes) for usage and porting information.

So the rest of that, after cloning, went like

$ pwd
/home/jubilee/rust/gcc
$ cd INSTALL
$ ls
README
$ cat README
This directory has been obsoleted for GCC snapshots and SVN access.

For releases the installation documentation is generated from
gcc/doc/install.texi and copied into this directory.

To read this documentation, please point your HTML browser to "index.html".
The latest version is always available at http://gcc.gnu.org/install/ .

$ ls ./index.html
ls: cannot access './index.html': No such file or directory
$ ls ../index.html
ls: cannot access '../index.html': No such file or directory
$ ls ../gcc/index.html
ls: cannot access '../gcc/index.html': No such file or directory
$ ls ../gcc/doc/index.html
ls: cannot access '../gcc/doc/index.html': No such file or directory
$ huh?
bash: huh?: command not found
$ cat README | grep index.html
To read this documentation, please point your HTML browser to "index.html".
$ cat ../gcc/doc/install.texi

...as for the output of that last one, uh, I probably should have used less there. Do the instructions for gcc's build require access to the online copy, which may have different instructions than the actual repo, in order to build the instructions to build the compiler, so I can read the current instructions to build the compiler?

I have built gcc before, but it was several years ago, and I distinctly remember thrashing quite a bit until a friend told me what the magic words were.

[saethlin]

One important thing to note is that currently the GCC backend is broken in the compiler

And yet CI currently passes. So what's going on here? Should we have an emergency patch to disable the GCC backend until the whole situation in this issue is resolved?

[workingjubilee]

And yes, coming back to the README and reviewing it more closely, I see it does describe how to build things under "Building with your own GCC version" but I did not want to do that, you see, I wanted to use your forked version instead? I assumed I could ignore this entire section because this if does not apply:

If you wrote a patch for GCC and want to test it without this backend, you will need to do a few more things.

I don't want to patch GCC, and I don't know what "without this backend" even means, honestly?

It is somewhat surprising to read instructions written for programming where conditionals that are logically false describe the code I actually want to have run? It doesn't seem that odd to want to simply build the code from source but at the same commit?

[GuillaumeGomez]

One important thing to note is that currently the GCC backend is broken in the compiler

And yet CI currently passes. So what's going on here? Should we have an emergency patch to disable the GCC backend until the whole situation in this issue is resolved?

It's not enabled anywhere and very few tests of the GCC backend are actually run in rustc's CI. It's mostly to ensure that it still builds for now.

[GuillaumeGomez]

I don't want to patch GCC, and I don't know what "without this backend" even means, honestly?

For reference, jubilee is talking about this. The section could get an improvement in its description since it's "our" gcc (already patched and everything) that we use everywhere. Also, I don't know what "without this backend" means either in this context. 😅

It is somewhat surprising to read instructions written for programming where conditionals that are logically false describe the code I actually want to have run? It doesn't seem that odd to want to simply build the code from source but at the same commit?

It's exactly what it's describing how to do: building gcc version with our patches so you can then test with the gcc backend.

[workingjubilee]

ah, I see! it seems the information is there but probably needs reframing so that it is no longer easy to interpret the condition as not being true?

[workingjubilee]

@GuillaumeGomez while using an x86_64-unknown-linux-gnu machine, after following the instructions to make, etc., I get this:

$ ./x.py test rustc_codegen_gcc
Building bootstrap
    Finished `dev` profile [unoptimized] target(s) in 0.02s
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.07s
Building compiler artifacts (stage0 -> stage1, x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.12s
Building stage0 codegen backend gcc (x86_64-unknown-linux-gnu)
    Finished `release` profile [optimized] target(s) in 0.01s
Creating a sysroot for stage1 compiler (use `rustup toolchain link 'name' build/host/stage1`)
    Finished `release` profile [optimized] target(s) in 0.07s
Building stage1 library artifacts (x86_64-unknown-linux-gnu)
error: process didn't exit successfully: `/home/jubilee/rust/rustc/build/bootstrap/debug/rustc -vV` (exit status: 1)
--- stdout
rustc 1.79.0-dev
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-gnu
release: 1.79.0-dev

--- stderr
error: couldn't load codegen backend /home/jubilee/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends/librustc_codegen_gcc-1.79.0-dev.so: libgccjit.so.0: cannot open shared object file: No such file or directory


Build completed unsuccessfully in 0:00:01
$ ls -lh | grep libgccjit
lrwxrwxrwx  1 jubilee jubilee   14 Apr 20 13:50 libgccjit.so -> libgccjit.so.0
lrwxrwxrwx  1 jubilee jubilee   18 Apr 20 13:50 libgccjit.so.0 -> libgccjit.so.0.0.1
-rwxr-xr-x  1 jubilee jubilee 305M Apr 20 13:50 libgccjit.so.0.0.1

Indeed, now that I am trying to load the gcc backend, everything seems to fail in this way. And setting $LIBRARY_PATH and $LD_LIBRARY_PATH does not seem to matter here? Is the only option here to learn yet another build system?

[GuillaumeGomez]

It's just that I don't know how well it would integrate with the current build system. In our own build system, it's very easy to use though

I mentioned it above. ^

[workingjubilee]

er, wait... didn't you add support to the current build system for it right here?

rust/src/bootstrap/src/core/build_steps/test.rs

Lines 3407 to 3531 in fb89862

	#[derive(Debug, Clone, PartialEq, Eq, Hash)]
	pub struct CodegenGCC {
	compiler: Compiler,
	target: TargetSelection,
	}
	impl Step for CodegenGCC {
	type Output = ();
	const DEFAULT: bool = true;
	const ONLY_HOSTS: bool = true;
	fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
	run.paths(&["compiler/rustc_codegen_gcc"])
	}
	fn make_run(run: RunConfig<'_>) {
	let builder = run.builder;
	let host = run.build_triple();
	let compiler = run.builder.compiler_for(run.builder.top_stage, host, host);
	if builder.doc_tests == DocTests::Only {
	return;
	}
	if builder.download_rustc() {
	builder.info("CI rustc uses the default codegen backend. skipping");
	return;
	}
	let triple = run.target.triple;
	let target_supported =
	if triple.contains("linux") { triple.contains("x86_64") } else { false };
	if !target_supported {
	builder.info("target not supported by rustc_codegen_gcc. skipping");
	return;
	}
	if builder.remote_tested(run.target) {
	builder.info("remote testing is not supported by rustc_codegen_gcc. skipping");
	return;
	}
	if !builder.config.codegen_backends(run.target).contains(&"gcc".to_owned()) {
	builder.info("gcc not in rust.codegen-backends. skipping");
	return;
	}
	builder.ensure(CodegenGCC { compiler, target: run.target });
	}
	fn run(self, builder: &Builder<'_>) {
	let compiler = self.compiler;
	let target = self.target;
	builder.ensure(compile::Std::new_with_extra_rust_args(
	compiler,
	target,
	&["-Csymbol-mangling-version=v0", "-Cpanic=abort"],
	));
	// If we're not doing a full bootstrap but we're testing a stage2
	// version of libstd, then what we're actually testing is the libstd
	// produced in stage1. Reflect that here by updating the compiler that
	// we're working with automatically.
	let compiler = builder.compiler_for(compiler.stage, compiler.host, target);
	let build_cargo = || {
	let mut cargo = builder::Cargo::new(
	builder,
	compiler,
	Mode::Codegen, // Must be codegen to ensure dlopen on compiled dylibs works
	SourceType::InTree,
	target,
	"run",
	);
	cargo.current_dir(&builder.src.join("compiler/rustc_codegen_gcc"));
	cargo
	.arg("--manifest-path")
	.arg(builder.src.join("compiler/rustc_codegen_gcc/build_system/Cargo.toml"));
	compile::rustc_cargo_env(builder, &mut cargo, target, compiler.stage);
	// Avoid incremental cache issues when changing rustc
	cargo.env("CARGO_BUILD_INCREMENTAL", "false");
	cargo.rustflag("-Cpanic=abort");
	cargo
	};
	builder.info(&format!(
	"{} GCC stage{} ({} -> {})",
	Kind::Test.description(),
	compiler.stage,
	&compiler.host,
	target
	));
	let _time = helpers::timeit(builder);
	// FIXME: Uncomment the `prepare` command below once vendoring is implemented.
	/*
	let mut prepare_cargo = build_cargo();
	prepare_cargo.arg("--").arg("prepare");
	#[allow(deprecated)]
	builder.config.try_run(&mut prepare_cargo.into()).unwrap();
	*/
	let mut cargo = build_cargo();
	cargo
	.arg("--")
	.arg("test")
	.arg("--use-system-gcc")
	.arg("--use-backend")
	.arg("gcc")
	.arg("--out-dir")
	.arg(builder.stage_out(compiler, Mode::ToolRustc).join("cg_gcc"))
	.arg("--release")
	.arg("--mini-tests")
	.arg("--std-tests");
	cargo.args(builder.config.test_args());
	let mut cmd: Command = cargo.into();
	builder.run_cmd(BootstrapCommand::from(&mut cmd).fail_fast());
	}
	}

and then PR the dev guide with the following command?

LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/12/ ./x.py test compiler/rustc_codegen_gcc/

leading me to input this, based on that, with a path to a freshly-built libgccjit:

$ cat config.toml
# Includes one of the default files in src/bootstrap/defaults
profile = "compiler"
change-id = 123711


[rust]
codegen-backends = ["llvm", "gcc"]

$ ls -lh "$(pwd)/../gcc-build/gcc" | grep "gccjit"
lrwxrwxrwx  1 jubilee jubilee    14 Apr 21 17:17 libgccjit.so -> libgccjit.so.0
lrwxrwxrwx  1 jubilee jubilee    18 Apr 21 17:17 libgccjit.so.0 -> libgccjit.so.0.0.1
-rwxr-xr-x  1 jubilee jubilee  305M Apr 21 17:17 libgccjit.so.0.0.1
$ LIBRARY_PATH="$(pwd)/../gcc-build/gcc" ./x.py test compiler/rustc_codegen_gcc

yet things don't work with my libgccjit built from the gcc fork on an x86-64 machine with afaik no custom nothing at this point? I'm... confused at my ability to break this, at this point. I am halfway to believing I have an ancient curse, a mutant electromagnetic field, or uncontrolled psionic abilities, because it seems like I have broken the "golden path" now.

[GuillaumeGomez]

It needs to have the libgccjit.so in your system library folder for now as far as I know. It's literally only used in the rustc's CI.

[bjorn3]

Would setting both LIBRARY_PATH and LD_LIBRARY_PATH work? The former is used by the linker running at compile time while the latter is used by the dynamic linker running at runtime.

[workingjubilee]

Would setting both LIBRARY_PATH and LD_LIBRARY_PATH work? The former is used by the linker running at compile time while the latter is used by the dynamic linker running at runtime.

I did try that, @bjorn3, and it seems it does not. :/

[saethlin]

If this backend is holding up development, in large part because developing on it appears to be impossible, why are we not discussing removing it until it becomes usable?

[antoyo]

If this backend is holding up development, in large part because developing on it appears to be impossible, why are we not discussing removing it until it becomes usable?

Is it really holding up development?

We never force people to implement the stuff for the GCC codegen: if they don't want to, they can just put unimplemented!() and we'll implement it on our side.
The CI won't even fail when doing that in most cases.

Please explain how this is holding up development.

[saethlin]

Ah! So the correct response in #124003 is indeed to just comment out or stub out failing tests. I probably just got confused because we seem to have very different standards for rustc_codegen_cranelift.

[GuillaumeGomez]

The current standing is different for both, cg_cranelift is much more advanced than cg_gcc. It's important for us to have it run as part of rustc's CI to greatly reduce the maintenance cost on our side.

[WaffleLapkin]

I'm a little late, but since antoyo invited me, here are my few cents.

From my perspective, as someone who touches the cg_ssa and only accidentally hits cg_gcc, I think that cg_gcc should be at least build-able (and maybe even testable) with common x.py commands. I.e. I want to be able to run x c rustc_codegen_gcc or x b rustc_codegen_gcc to make sure that it builds, locally. Ideally it should just work out of the box, without any other setup. libgccjit should either be downloaded, or built by the bootstrap too (it should be configurable similarly to llvm). Ideally it should work on both x86_64 and aarch64 (at least since the latter is the arch of some dev desktops).

This should make it much easier to do changes to cg_gcc when they are needed collaterally (e.g. as was in #124003). Because currently, the setup steps are too convoluted to follow.

[GuillaumeGomez]

I was planning to make x.py handle the download of libgccjit.so (for x86_64) since it should now be in the rust CI's artifacts. Just didn't get to it yet. Gonna resume this to make everyone's life easier.

[GuillaumeGomez]

I opened #124353. It should fix most issues. For now it only supports linux x86_64 target. We can expand from there pretty easily.

@workingjubilee: Your problem was that we did not set the LD_LIBRARY_PATH environment variable, so it wasn't in your path, hence why the build system couldn't find it.