New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start using serde_derive in a couple places in the compiler. #56447

Open
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
7 participants
@eddyb
Member

eddyb commented Dec 2, 2018

Note that this doesn't actually use serde for anything currently - I have a different branch where I started that, but I thought I'd extract this proof of concept out, to deal with build system issues.

The second commit is needed to fix this error in rustc_codegen_llvm and rustdoc:

error[E0463]: can't find crate for `serde_derive` which `rustc` depends on

This problem arises because we use different Cargo build dirs for codegen backends and rustdoc from the main rustc Cargo build dir, and without the second commit in this PR, the host deps (including serde_derive) weren't copied over to the common sysroot (shared between all builds).

Quite surprisingly, everything else seems to "just work"!
This is in part because of pre-existing logic in rustbuild to only use stage0 for build scripts when building libstd itself, which happens to be what we need to make proc macros work in all stages 馃帀

cc @alexcrichton @Mark-Simulacrum @Zoxc @nikomatsakis

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Dec 2, 2018

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

@eddyb

This comment has been minimized.

Member

eddyb commented Dec 2, 2018

@Mark-Simulacrum

This comment has been minimized.

Member

Mark-Simulacrum commented Dec 2, 2018

@bors try to check a dist build (though I don't expect it to be different)

@bors

This comment has been minimized.

Contributor

bors commented Dec 2, 2018

鈱涳笍 Trying commit 7b22fcb with merge d934074...

bors added a commit that referenced this pull request Dec 2, 2018

Auto merge of #56447 - eddyb:serde-poc, r=<try>
Start using serde_derive in a couple places in the compiler.

Note that this doesn't actually use `serde` for anything currently - I have a different branch where I started that, but I thought I'd extract this proof of concept out, to deal with build system issues.

The second commit is needed to fix this error in `rustc_codegen_llvm` and `rustdoc`:
```
error[E0463]: can't find crate for `serde_derive` which `rustc` depends on
```
This problem arises because we use different Cargo build dirs for codegen backends and `rustdoc` from the main `rustc` Cargo build dir, and without the second commit in this PR, the host `deps` (including `serde_derive`) weren't copied over to the common sysroot (shared between all builds).

*Quite surprisingly*, everything else seems to "just work"!
This is in part because of pre-existing logic in `rustbuild` to only use `stage0` for build scripts when building libstd itself, which *happens to* be what we need to make proc macros work in all stages 馃帀

cc @alexcrichton @Mark-Simulacrum @Zoxc @nikomatsakis
@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Dec 3, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:21b204be:start=1543791871065679542,finish=1543791926927771954,duration=55862092412
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:59:30] 
[00:59:30] running 119 tests
[00:59:33] i..ii...iii..iiii.....i...i.........i..iii.............i......i....ii...i..i.ii..............i...ii. 100/119
[00:59:33] .ii.i.....iiii.....
[00:59:33] 
[00:59:33]  finished in 3.559
[00:59:33] travis_fold:end:test_codegen

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:59:48] 
[00:59:48] running 118 tests
[01:00:13] .iiiii...i.....i..i...i..i.i..i.i..i.....i..i....i..........iiii.........i.i....i...i.......ii.i.i.i 100/118
[01:00:17] ......iii.i.....ii
[01:00:17] 
[01:00:17]  finished in 29.105
[01:00:17] travis_fold:end:test_debuginfo

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@bors

This comment has been minimized.

Contributor

bors commented Dec 3, 2018

鈽锔 Test successful - status-travis
State: approved= try=True

@rust-highfive

This comment has been minimized.

Collaborator

rust-highfive commented Dec 3, 2018

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
travis_time:end:33bc9f16:start=1543817953872979004,finish=1543818009818555471,duration=55945576467
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#Pull-Requests-and-Security-Restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-5.0
---
travis_time:start:test_codegen
Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:57:00] 
[00:57:00] running 119 tests
[00:57:03] i..ii...iii..iiii.....i...i.........i..iii.............i.....i.....ii...i..i.ii..............i...ii. 100/119
[00:57:04] .ii.i.....iiii.....
[00:57:04] 
[00:57:04]  finished in 3.356
[00:57:04] travis_fold:end:test_codegen

---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:57:18] 
[00:57:18] running 118 tests
[00:57:41] .iiiii...i.....i..i...i..i.i..i.i..i.....i..i....i..........iiii.........i.i....i...i.......ii.i.i.i 100/118
[00:57:45] ......iii.i.....ii
[00:57:45] 
[00:57:45]  finished in 27.413
[00:57:45] travis_fold:end:test_debuginfo

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@eddyb

This comment has been minimized.

Member

eddyb commented Dec 3, 2018

Looks like the last failure is caused by build_helper and cmake being copied, with the rest of the host-specific deps, but they're built by stage0/bin/rustc.

Also, in general, I feel like we should distinguish between "crates you can use from the sysroot" and "dependencies of sysroot crates", similar to how Cargo uses -Ldependency for deps.
Maybe have a directory nested in, or next to, lib/rustlib/$target/lib, for such deps.
That way, people can't just accidentally, say, use extern crate serde; and get an error from rustc_private (even if we provide a useful message, it's still suboptimal).

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Dec 3, 2018

Hm I think I'm a bit lost here, how does this patch work? Shouldn't procedural macros always be compiled by the compiler that build the standard library in that stage? (aka not always the stage0 compiler). I think though that with our shim it's always using the stage0 compiler?

Also.. how does this work without your patches on nightly for decoupling proc_macro/rustc?

@eddyb

This comment has been minimized.

Member

eddyb commented Dec 4, 2018

@alexcrichton So the helpfully-preexisting behavior I allude in the PR description is here:

// Almost all of the crates that we compile as part of the bootstrap may
// have a build script, including the standard library. To compile a
// build script, however, it itself needs a standard library! This
// introduces a bit of a pickle when we're compiling the standard
// library itself.
//
// To work around this we actually end up using the snapshot compiler
// (stage0) for compiling build scripts of the standard library itself.
// The stage0 compiler is guaranteed to have a libstd available for use.
//
// For other crates, however, we know that we've already got a standard
// library up and running, so we can use the normal compiler to compile
// build scripts in that situation.
//
// If LLVM support is disabled we need to use the snapshot compiler to compile
// build scripts, as the new compiler doesn't support executables.
if mode == Mode::Std || !self.config.llvm_enabled {
cargo
.env("RUSTC_SNAPSHOT", &self.initial_rustc)
.env("RUSTC_SNAPSHOT_LIBDIR", self.rustc_snapshot_libdir());
} else {
cargo
.env("RUSTC_SNAPSHOT", self.rustc(compiler))
.env("RUSTC_SNAPSHOT_LIBDIR", self.rustc_libdir(compiler));
}

That's almost literally the exact same logic I mentioned I was considering implementing on Discord, except someone already did it!
Build scripts can always use stage0/bin/rustc so they don't need it, but it still applies.

As for stage0/bin/rustc not supporting the new decoupling: we can't use stage0/bin/rustc with the locally built src/libproc_macro anyway, while src/libproc_macro is now "ABI-safe", it still needs two copies built from the same source to be compatible (we could even include a version hash in the protocol to ensure we don't accidentally mix them up).

What should happen at stage0 is that proc macros are built against (the downloaded) stage0/lib/rustlib/**/libproc_macro*, not the locally-built version, but higher stages should use the proc_macro they built (from stage1/lib/rustlib).

Note that using proc macros in the compiler has always worked, but only at stage0, my decoupling fixes stage1 onwards (but primarily only stage1).

This might be accidentally right in rustbuild, again, because the locally-built libraries of stage0 are in stage0-sysroot/lib/rustlib, while stage0 contains the downloaded versions.
But stageN/lib/rustlib where N > 0 contains locally-built-by-stageN libraries, so what we need happens to be in stageN/lib/rustlib for all N.


To be really honest, I wasn't expecting this to work either without more tweaks. But someone from the past was too clever in rustbuild and stumbled upon what we need for proc macros!

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Dec 4, 2018

Ok this is starting to make a bit more sense I think, we'll always compile procedural macros against the stage0 proc_macro crate, and it's the proc_macro crate freshly compiled.

What I still don't think I understand though is how this works at all in stage0 right now. Currently our stage0 compiler does the old procedural macro system. This PR, however, compiles serde_derive in stage0 against the stage0 proc_macro crate previously built. That proc_macro crate is wildly incompatible with the bootstrap compiler which we're compiling with, right? How does that get hooked up correctly?

As a side note as well, does this mean that we can never change the ABI of procedural macros? If procedural macros are always compiled against the stage0 version, how do we change the signatures?

// If this was output in the `deps` dir then this is a precise file
// name (hash included) so we start tracking it.
if filename.starts_with(&target_deps_dir) {
if filename.starts_with(&host_root_dir) || filename.starts_with(&target_deps_dir) {

This comment has been minimized.

@alexcrichton

alexcrichton Dec 4, 2018

Member

Can you expand a bit on what's going on here? This feels like it shouldn't be necessary...

This comment has been minimized.

@eddyb

eddyb Dec 4, 2018

Member

The PR description tries to explain this. It's needed for the compiler to be able to find the (host-only) proc macro dependencies of e.g. librustc, from across a Cargo build (i.e. from codegen backends).

This comment has been minimized.

@alexcrichton

alexcrichton Dec 7, 2018

Member

Hm ok, I ask because I don't think that this is right. This I think is definitely not what we want when cross-compiling because it's mixing architectures of libraries in one directory. For non-cross-compiling situations this'll copy too much for things like build scripts and/or intermediate proc-macro dependencies.

Could we perhaps whitelist an explicit copy of serde_derive if necessary? Or something like that?

This comment has been minimized.

@Zoxc

Zoxc Dec 7, 2018

Contributor

It would be nice if cargo would tell us if artifacts are proc macros, but I think a whitelist will suffice here

This comment has been minimized.

@eddyb

eddyb Dec 7, 2018

Member

Mixing architectures isn't a problem for rustc, and a whitelist feels hackier IMO.

I still like the idea of making the same -Ldependency= distinction Cargo makes, in the sysroot, as well (from #56447 (comment)).

In fact, we can even copy the host deps to rustlib/$host/deps instead of rustlib/$target/deps, and have the compiler look in the right place.


I should also mention I was able to work around this by adding a serde_derive dependency to librustc_codegen_llvm, but I feel like this also doesn't scale (and it builds serde_derive more than once).


I guess I'd accept the whitelisting if someone else implemented it, just to unblock @Zoxc.

@bors

This comment has been minimized.

Contributor

bors commented Dec 8, 2018

鈽旓笍 The latest upstream changes (presumably #56578) made this pull request unmergeable. Please resolve the merge conflicts.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Dec 12, 2018

@eddyb do you perhaps want to take the strategy from #56462 to rebase and land this?

bors added a commit that referenced this pull request Dec 14, 2018

Auto merge of #56795 - Zoxc:serde-poc, r=alexcrichton
Start using serde_derive in a couple places in the compiler

This is #56447 with the build changes in #56462

r? @alexcrichton
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment