New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support producing a "build plan" without executing anything #3815

Closed
alexcrichton opened this Issue Mar 10, 2017 · 13 comments

Comments

Projects
None yet
10 participants
@alexcrichton
Copy link
Member

alexcrichton commented Mar 10, 2017

This is an issue extracted from the discussion over at rust-lang/rust-roadmap-2017#12. The general high-level idea is that Cargo should be able to produce a build plan for the explicit purpose of consumption by another tool. This step would not iteslf execute any work but will likely assume that all the source code is available for consumption.

Once Cargo is able to support this it should be explored to see what this integration would then look like into external build system, such as Buck or Bazel. This may involve writing generators which translates from Cargo's build plan to a Buck/Bazel configuration file, or it may involve more dynamism at build time.

The goal here is to leverage Cargo as much as possible for drawing dependencies between projects and learning how the compiler is executed. This should give us quite a bit of flexibility to continue to implement new features in Cargo (such as the recently stabilized proc-macro) with little-to-no changes in external build tools.

@P-E-Meunier

This comment has been minimized.

Copy link

P-E-Meunier commented Apr 9, 2017

I love the idea, and I actually wrote such an external tool relying on Nix, see https://nest.pijul.com/pmeunier/nix-rust

Just parsing Cargo.lock was enough to get reproducible builds on NixOS, without recompiling the world every time. However, just two things still feel hackish:

  • I had to guess what the main file of a crate was, and whether the crate was a binary or a library. For some crates (such as "untrusted"), the Cargo.toml simply says lib.name = "untrusted", and there's a single file in the package, also called "untrusted".

  • Features. Nix computes a hash of everything used to produce a package, so it can track feature changes easily.

Including these two things in the lock file would solve everything!

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Apr 10, 2017

Some more notes from various discussions:

  • This output should include all inputs necessary to produce such output. Basically this should imply running --emit dep-info on the compiler to get a list of input files
  • The output here should be a full dependency graph for everything on the Cargo side of things, with edges between crates everywhere.
  • The output should also also list all the outputs of each step, rlibs/dylibs/etc.
@bennofs

This comment has been minimized.

Copy link
Contributor

bennofs commented Jun 29, 2017

Clang/LLVM has something similar: build_commands.json

@glandium

This comment has been minimized.

Copy link

glandium commented Jul 6, 2017

FWIW, the compilation database for clang made a big mistake where one needs to parse the command on their own. It should have been a list instead of a string. Please don't repeat that mistake.

@da-x

This comment has been minimized.

Copy link
Member

da-x commented Jul 6, 2017

This feature is akin to gcc's -M family of flags. For the purity of the solution I suggest to add the following:

  • Support for non-existing inputs, i.e. input .rs files that are supposed to be generated prior to the execution of the actual rustc invocation. E.g. outputs of larlpop. Perhaps these external tools could be integrated in the output build-plan so that it covers a greater scope?
  • Build plan should not depend on any prior built state (i.e. the target/ directory need not exist and is not read for the build plan to be generated).
@budziq

This comment has been minimized.

Copy link

budziq commented Jul 13, 2017

It would be ideal if such functionality could be some how triggered/available while in build.rs.
Then relatively hacky testing crates such rust-skeptic could stop parsing unstable cargo .fingerprint internals while building documentation testcases 😉
budziq/rust-skeptic#37

mshal added a commit to mshal/cargo that referenced this issue Aug 3, 2017

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
Python data structures. The 'dependencies' structure lists the
dependencies between steps, and the 'commands' structure lists the
information required to run the commands from an external build system.

Fixes rust-lang#3815
@mshal

This comment has been minimized.

Copy link
Contributor

mshal commented Aug 3, 2017

I made an attempt at implementing this as a way of learning Rust. So far it just prints out the info from DependencyQueue and the rustc command-line invocations as Python data structures. With this I'm able to import the data structures into a Python program that writes out Tupfiles, and build a small dummy Rust program (a hello-world with two local library dependencies) with tup.

There are a number of things missing so far, so this isn't ready to land. In particular:

  1. build.rs support is not yet implemented. Confusingly, it will output a build step for a build.rs file with the same name as the rustc invocation.
  2. The final output of a program is not copied. It will create a rule for target/debug/deps/hello_world-b6a758bc870b5332, but not target/debug/hello_world, for example.
  3. Running cargo with --build-plan still creates some directories under target/. This should probably be left to the external build system.
@aturon

This comment has been minimized.

Copy link
Member

aturon commented Aug 3, 2017

Just a quick note -- the Cargo team is actively hashing out high-level design questions around this and other aspects of build system integration. Hope to have some writeups soon!

@Xanewok

This comment has been minimized.

Copy link
Member

Xanewok commented Aug 14, 2017

@mshal awesome that you did some work on this!
If possible, I'd like to help with the implementation as it'd be very convenient to have that in the RLS, since right now we're going to use an approximation to perform our own builds using rustc.

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Sep 2, 2017

There's an RFC for build system integration now up at rust-lang/rfcs#2136

mshal added a commit to mshal/cargo that referenced this issue Nov 17, 2017

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes rust-lang#3815
@mshal

This comment has been minimized.

Copy link
Contributor

mshal commented Nov 17, 2017

Sorry for the delay - I've been distracted by other things. I have the latest attempt based off of cargo-0.22 here: https://github.com/mshal/cargo/tree/build-plan-0.22.0

I used 0.22 as the base because that's what we currently use in mozilla-central. I'm able to run cargo with --build-plan against the toolkit/library/rust/Cargo.toml file to generate a plan.json which contains (almost) all the information needed to invoke the rustc commands and build scripts with an external tool. The only missing thing I am aware of is the outputs of a build script, but those can be supplemented externally as a workaround.

It almost works when I run it through tup, but that has other problems with rustc's weird file usage patterns that still need to be resolved. I believe this is a reasonably complete implementation in the cargo side of things however.

I'd appreciate any feedback and improvements! If it would be easier, I could submit it as a pull request.

@mshal

This comment has been minimized.

Copy link
Contributor

mshal commented Nov 17, 2017

I forgot to mention, but I did convert the json file to a shell script that just invokes the rustc commands in the right order, and that shell script does complete successfully. So I think that shows it can at least function as a way of exporting commands to an external "build system"

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Nov 20, 2017

Awesome thanks for the update @mshal! A PR would probably be the easiest thing for review, yeah, so feel free!

mshal added a commit to mshal/cargo that referenced this issue Mar 27, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes rust-lang#3815

mshal added a commit to mshal/cargo that referenced this issue Apr 5, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes rust-lang#3815

mshal added a commit to mshal/cargo that referenced this issue Apr 23, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes rust-lang#3815

mshal added a commit to mshal/cargo that referenced this issue May 1, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes rust-lang#3815

mshal added a commit to mshal/cargo that referenced this issue May 3, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes rust-lang#3815

mshal added a commit to mshal/cargo that referenced this issue May 7, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes rust-lang#3815

bors added a commit that referenced this issue May 10, 2018

Auto merge of #5301 - mshal:build-plan, r=matklad
Add --build-plan for 'cargo build'

With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815

@bors bors closed this in #5301 May 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment