Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement "pipelined" rustc compilation (compiler side) #58465

Closed
nikomatsakis opened this issue Feb 14, 2019 · 12 comments

Comments

@nikomatsakis
Copy link
Contributor

commented Feb 14, 2019

@alexcrichton opened rust-lang/cargo#6660, which describes a scheme to do "pipelined" compilation. This issue represents the rustc side of that work. I'm opening it -- initially, at least -- so that I can nominate the issue and we can discuss in the compiler team meetings. It may then serve as the hub for any mentoring re: the rustc part of the equation.

I would like us to decide whether to do this work and how to integrate that into our overall planning (who will do the rustc part of the work? How hard is it? etc)

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Apr 2, 2019

For the rustc side of things I believe there's two primary work items to implement:

  • The compiler should be able to produce an rlib when the dependencies are only available as rmeta files. Currently when producing an rlib I believe the compiler requires all dependencies to be an rlib, but all dependencies only need to be an rlib if we're producing a final linkable artifact. When an rlib is produced only the metadata needs to be fundamentally available for all dependencies.

  • Somehow the compiler needs to signal to Cargo when the rmeta file for a compilation is ready to go. The rmeta file would be produced as soon as possible in compilation and then Cargo would be signaled to its existence so Cargo could continue compilation of later dependencies if necessary. The exact mechanism to do this is debated in rust-lang/cargo#6660, but the current to main candidates are: send a message to a TCP server hosted by Cargo, or print something to stdout which Cargo will read

@ehuss

This comment has been minimized.

Copy link
Contributor

commented Apr 2, 2019

Currently when producing an rlib I believe the compiler requires all dependencies to be an rlib

I'm able to get around this with -Z always-encode-mir (it seems to successfully create the rlib from rmeta deps), but when linking the final output with these rlibs, I get crate foo required to be available in rlib format, but was not found in this form. Is there a way to tell rustc how to pick up the rlibs for transitive deps? Or is that embedded in the rlib with no way to override?

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Apr 2, 2019

I think that -Z always-encode-mir being required is a bug and I suspect that the other error message is likely indicative of the same bug. While it may be possible to work around, I think we'll definitely want to change rustc and how it loads dependencies to fully fix this. I think today it's basically not possible to emulate the behavior we want

@AGausmann

This comment has been minimized.

Copy link

commented Apr 2, 2019

My first instinct would be to make rmeta generation a subcommand of the compiler, with the signal being the termination of the process. After that, the compiler can be re-invoked to create the rlib. The feasibility of that would depend on how much compiler state gets stored to disk during rmeta creation, and how much would get lost on exit that would have to be recreated on the second run, which I don't know much about.

@nnethercote

This comment has been minimized.

Copy link
Contributor

commented Apr 2, 2019

The produce-an-rlib-from-rmeta-inputs part needs to be done first and sounds like the larger piece of work. I would like to work on that.

rust-lang/cargo#6660 has some discussion of the cargo/rustc interaction, pros and cons of different approaches.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Apr 4, 2019

So it turns out I was remember a time long since passed, and my first bullet above is seemingly already done (@nnethercote is going to fully confirm). That means that actually all rustc needs to do is to somehow signal Cargo when a file is ready to go!

I've talked with @nnethercote about this and I think he's going to work on implementing the JSON message idea (where rustc emits a JSON message whenever a file is ready for Cargo to consume).

@ehuss FWIW rustc generates different rmeta files if you do --emit metadata vs if you do --emit metadata,link, so I think that's why you needed -Z always-encode-mir. Cargo would, however, always execute --emit metadata,link,dep-info, so I think we're covered!

@ehuss

This comment has been minimized.

Copy link
Contributor

commented Apr 4, 2019

Oh, indeed, it does seem to work!

@pnkfelix

This comment has been minimized.

Copy link
Member

commented Apr 11, 2019

There is now WG-pipelining: https://github.com/rust-lang/compiler-team/tree/master/working-groups/pipelining

I'm going to remove the nominated tag from this ticket.

I don't know whether this ticket should be repurposed into a tracking issue for this work, or simply be closed. I'll leave that choice up to WG-pipelining.

@pnkfelix pnkfelix removed the I-nominated label Apr 11, 2019

@nnethercote

This comment has been minimized.

Copy link
Contributor

commented Apr 30, 2019

#60006 just landed, which implements the JSON message rustc gives to Cargo so that Cargo knows when metadata has been written.

I'm now working on #60385, which will move metadata writing much earlier in the pipeline -- currently it happens right at the end, preventing much in the way of crate overlap that is the whole goal of pipelining.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented Apr 30, 2019

I've opened a dedicated tracking issue for stabilizing -Z emit-directives

@alexcrichton

This comment has been minimized.

Copy link
Member

commented May 17, 2019

I've made a post on internals about evaluating pipelined compilation now that nightly Cargo/rustc both fully support pipelined compilation.

@alexcrichton

This comment has been minimized.

Copy link
Member

commented May 20, 2019

Since this issue as listed is done, I'm going to close this in favor of a new tracking issue for stabilization - #60988

Centril added a commit to Centril/rust that referenced this issue May 22, 2019

Rollup merge of rust-lang#61014 - jsgf:emit-artifact-type, r=alexcric…
…hton

Make -Zemit-artifact-notifications also emit the artifact type

This is easier for tooling to handle than trying to reverse-engineer the type from the filename extension. The field name and value is intended to reflect the `--emit` command-line option.

Related issues rust-lang#60988 rust-lang#58465
cc @alexcrichton

Centril added a commit to Centril/rust that referenced this issue May 23, 2019

Rollup merge of rust-lang#61014 - jsgf:emit-artifact-type, r=alexcric…
…hton

Make -Zemit-artifact-notifications also emit the artifact type

This is easier for tooling to handle than trying to reverse-engineer the type from the filename extension. The field name and value is intended to reflect the `--emit` command-line option.

Related issues rust-lang#60988 rust-lang#58465
cc @alexcrichton
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.