clippy_lints is breaking the stage1 build

[pnkfelix]

Atop commit f6d7514 , I am trying to build Rust. I have deleted my build/ subdirectory, and I have even attempted to build from the root directory (rust.git), even though I usually put specific config.toml into subdirectories (to maintain separate debug and release builds)...

But on both Linux and Mac OS X, I am getting the same basic end result when I try to build:

% ./x.py test src/tools/tidy && RUST_BACKTRACE=1  ./x.py build --stage 1 && ./x.py test --stage 1 src/test/{mir-opt,compile-fail,run-pass}
[...]
   Compiling serde_derive_internals v0.16.0
   Compiling serde_derive v1.0.15
   Compiling clippy_lints v0.0.165 (file:///home/pnkfelix/Dev/Mozilla/rust.git/src/tools/clippy/clippy_lints)
   Compiling cargo_metadata v0.2.3
error: libproc_macro-6210e4b46662ec28.so: cannot open shared object file: No such file or directory
  --> /home/pnkfelix/.cargo/registry/src/github.com-1ecc6299db9ec823/cargo_metadata-0.2.3/src/lib.rs:13:14
   |
13 | #[macro_use] extern crate serde_derive;
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^

error: libproc_macro-6210e4b46662ec28.so: cannot open shared object file: No such file or directory
  --> /home/pnkfelix/.cargo/registry/src/github.com-1ecc6299db9ec823/cargo_metadata-0.2.3/src/lib.rs:13:14
   |
13 | #[macro_use] extern crate serde_derive;
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile `cargo_metadata`.
warning: build failed, waiting for other jobs to finish...
error: Could not compile `cargo_metadata`.
warning: build failed, waiting for other jobs to finish...
error: libproc_macro-6210e4b46662ec28.so: cannot open shared object file: No such file or directory
  --> src/tools/clippy/clippy_lints/src/lib.rs:47:1
   |
47 | extern crate serde_derive;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^

error: libproc_macro-6210e4b46662ec28.so: cannot open shared object file: No such file or directory
  --> src/tools/clippy/clippy_lints/src/lib.rs:47:1
   |
47 | extern crate serde_derive;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile `clippy_lints`.
warning: build failed, waiting for other jobs to finish...
error: Could not compile `clippy_lints`.

To learn more, run the command again with --verbose.


command did not execute successfully: "/home/pnkfelix/Dev/Mozilla/rust.git/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "8" "--release" "--manifest-path" "/home/pnkfelix/Dev/Mozilla/rust.git/src/tools/clippy/Cargo.toml"

(On Mac OS X it instead complains:

error: dlopen(/Users/fklock/Dev/Mozilla/rust-mirborrowck/build/x86_64-apple-darwin/stage1-tools/release/deps/libserde_derive-c43f7547bcf0ad68.dylib, 1): Library not loaded: @rpath/libproc_macro-4a190002959d1e04.dylib
  Referenced from: /Users/fklock/Dev/Mozilla/rust-mirborrowck/build/x86_64-apple-darwin/stage1-tools/release/deps/libserde_derive-c43f7547bcf0ad68.dylib
  Reason: image not found
  --> /Users/fklock/.cargo/registry/src/github.com-1ecc6299db9ec823/cargo_metadata-0.2.3/src/lib.rs:13:14
   |
13 | #[macro_use] extern crate serde_derive;
   |              ^^^^^^^^^^^^^^^^^^^^^^^^^^

but I interpret that as essentially the same error.

[pnkfelix]

cc @Manishearth @llogiq @mcarton @oli-obk

[pnkfelix]

cc #45177

[kennytm]

@pnkfelix Could you show your config.toml, if any, and how the x.py command is executed?

[pnkfelix]

@kennytm yeah I just realized I should add that info.

I added the command invocation to the transcript. There is no config.toml in my root directory.

The relevant part of the command invocation, I think, is: RUST_BACKTRACE=1 ./x.py build --stage 1

[kennytm]

@pnkfelix thanks!

---

cc #44629 — building compiler doc also failed by "libproc_macro" not found, may be relevant.

[pnkfelix]

ah so this might be specific to stage 1. I'll add that to the title

[Manishearth]

This isn't a clippy issue, this is a bug with rustc/cargo wrt dylibs that's been around for ages. It's only cropping up now more because custom derives are a thing (but it always used to be a problem with plugins)

[Manishearth]

This same issue breaks the internals rustdoc build.

[pnkfelix]

@Manishearth its a clippy issue if clippy being added to our tree breaks the work flow for stage1 builds. :(

[Manishearth]

Yes, I mean that this isn't a bug that can be fixed in clippy; it's a longstanding bug with dylib loading with rustc.

[arielb1]

So this is a rustbuild bug - we should just avoid building clippy in stage1. Or we should ship a blessed version of serde_derive as a part of stage0 and make clippy use it but that would suck.

[kennytm]

It is also breaking stage0. I agree with @arielb1, build without any path should only build src/librustc and src/tools/rustdoc, nothing more.

@pnkfelix You could workaround by explicitly passing the path, i.e. ./x.py build --stage 1 src/librustc.

[pnkfelix]

Should the ToolStates structure be a mapping (tool, stage) -> ToolState instead of just a mapping from tool -> ToolState? I'm trying to figure out the right way to represent this knowledge about which tools can be built via the stage1 rustc vs which ones require the stage2 rustc.

[oli-obk]

I'm trying to figure out the right way to represent this knowledge about which tools can be built via the stage1 rustc vs which ones require the stage2 rustc.

I thought

rust/src/bootstrap/tool.rs

Lines 405 to 418 in f6d7514

	// Clippy depends on procedural macros (serde), which requires a full host
	// compiler to be available, so we need to depend on that.
	builder.ensure(compile::Rustc {
	compiler: self.compiler,
	target: builder.build.build,
	});
	builder.ensure(ToolBuild {
	compiler: self.compiler,
	target: self.target,
	tool: "clippy-driver",
	mode: Mode::Librustc,
	path: "src/tools/clippy",
	expectation: builder.build.config.toolstate.clippy.passes(ToolState::Compiling),
	})

already took care of that, since that is copy pasted from cargo.

I'm not sure why it isn't working as advertised...

[Mark-Simulacrum]

Clippy, Cargo, and Rustdoc currently must be built with a stage 2 compiler due to how proc macros work. I don't think there's anything rustbuild can do in particular to change that, other than not building them in earlier stages (which does seem like a good idea).

[Manishearth]

Can we also exclude stage 1 from the doc build while we're at it? The doc build has been broken for months because cssparser uses a proc macro.

[sanmai-NL]

@kennytm is this issue ready for closing, post #45548?