Support producing a "build plan" without executing anything #3815

Closed
alexcrichton opened this Issue Mar 10, 2017 · 13 comments

Comments

Projects
None yet
10 participants
@alexcrichton
Member

alexcrichton commented Mar 10, 2017

This is an issue extracted from the discussion over at rust-lang/rust-roadmap-2017#12. The general high-level idea is that Cargo should be able to produce a build plan for the explicit purpose of consumption by another tool. This step would not iteslf execute any work but will likely assume that all the source code is available for consumption.

Once Cargo is able to support this it should be explored to see what this integration would then look like into external build system, such as Buck or Bazel. This may involve writing generators which translates from Cargo's build plan to a Buck/Bazel configuration file, or it may involve more dynamism at build time.

The goal here is to leverage Cargo as much as possible for drawing dependencies between projects and learning how the compiler is executed. This should give us quite a bit of flexibility to continue to implement new features in Cargo (such as the recently stabilized proc-macro) with little-to-no changes in external build tools.

@P-E-Meunier

This comment has been minimized.

Show comment
Hide comment
@P-E-Meunier

P-E-Meunier Apr 9, 2017

I love the idea, and I actually wrote such an external tool relying on Nix, see https://nest.pijul.com/pmeunier/nix-rust

Just parsing Cargo.lock was enough to get reproducible builds on NixOS, without recompiling the world every time. However, just two things still feel hackish:

  • I had to guess what the main file of a crate was, and whether the crate was a binary or a library. For some crates (such as "untrusted"), the Cargo.toml simply says lib.name = "untrusted", and there's a single file in the package, also called "untrusted".

  • Features. Nix computes a hash of everything used to produce a package, so it can track feature changes easily.

Including these two things in the lock file would solve everything!

P-E-Meunier commented Apr 9, 2017

I love the idea, and I actually wrote such an external tool relying on Nix, see https://nest.pijul.com/pmeunier/nix-rust

Just parsing Cargo.lock was enough to get reproducible builds on NixOS, without recompiling the world every time. However, just two things still feel hackish:

  • I had to guess what the main file of a crate was, and whether the crate was a binary or a library. For some crates (such as "untrusted"), the Cargo.toml simply says lib.name = "untrusted", and there's a single file in the package, also called "untrusted".

  • Features. Nix computes a hash of everything used to produce a package, so it can track feature changes easily.

Including these two things in the lock file would solve everything!

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Apr 10, 2017

Member

Some more notes from various discussions:

  • This output should include all inputs necessary to produce such output. Basically this should imply running --emit dep-info on the compiler to get a list of input files
  • The output here should be a full dependency graph for everything on the Cargo side of things, with edges between crates everywhere.
  • The output should also also list all the outputs of each step, rlibs/dylibs/etc.
Member

alexcrichton commented Apr 10, 2017

Some more notes from various discussions:

  • This output should include all inputs necessary to produce such output. Basically this should imply running --emit dep-info on the compiler to get a list of input files
  • The output here should be a full dependency graph for everything on the Cargo side of things, with edges between crates everywhere.
  • The output should also also list all the outputs of each step, rlibs/dylibs/etc.

@Ericson2314 Ericson2314 referenced this issue in rust-lang/rfcs Apr 30, 2017

Closed

Add Cargo post-build scripts #1777

@sdboyer sdboyer referenced this issue in golang/dep May 11, 2017

Open

Native cgo support infrastructure #269

@bennofs

This comment has been minimized.

Show comment
Hide comment
@bennofs

bennofs Jun 29, 2017

Contributor

Clang/LLVM has something similar: build_commands.json

Contributor

bennofs commented Jun 29, 2017

Clang/LLVM has something similar: build_commands.json

@glandium

This comment has been minimized.

Show comment
Hide comment
@glandium

glandium Jul 6, 2017

FWIW, the compilation database for clang made a big mistake where one needs to parse the command on their own. It should have been a list instead of a string. Please don't repeat that mistake.

glandium commented Jul 6, 2017

FWIW, the compilation database for clang made a big mistake where one needs to parse the command on their own. It should have been a list instead of a string. Please don't repeat that mistake.

@da-x

This comment has been minimized.

Show comment
Hide comment
@da-x

da-x Jul 6, 2017

Member

This feature is akin to gcc's -M family of flags. For the purity of the solution I suggest to add the following:

  • Support for non-existing inputs, i.e. input .rs files that are supposed to be generated prior to the execution of the actual rustc invocation. E.g. outputs of larlpop. Perhaps these external tools could be integrated in the output build-plan so that it covers a greater scope?
  • Build plan should not depend on any prior built state (i.e. the target/ directory need not exist and is not read for the build plan to be generated).
Member

da-x commented Jul 6, 2017

This feature is akin to gcc's -M family of flags. For the purity of the solution I suggest to add the following:

  • Support for non-existing inputs, i.e. input .rs files that are supposed to be generated prior to the execution of the actual rustc invocation. E.g. outputs of larlpop. Perhaps these external tools could be integrated in the output build-plan so that it covers a greater scope?
  • Build plan should not depend on any prior built state (i.e. the target/ directory need not exist and is not read for the build plan to be generated).

@luser luser referenced this issue in rust-lang-nursery/rls Jul 6, 2017

Closed

Don't require the build_lib flag #332

@Xanewok Xanewok referenced this issue in rust-lang-nursery/rls Jul 8, 2017

Open

Integrate with large (non-cargo) build systems #401

@budziq

This comment has been minimized.

Show comment
Hide comment
@budziq

budziq Jul 13, 2017

It would be ideal if such functionality could be some how triggered/available while in build.rs.
Then relatively hacky testing crates such rust-skeptic could stop parsing unstable cargo .fingerprint internals while building documentation testcases 😉
budziq/rust-skeptic#37

budziq commented Jul 13, 2017

It would be ideal if such functionality could be some how triggered/available while in build.rs.
Then relatively hacky testing crates such rust-skeptic could stop parsing unstable cargo .fingerprint internals while building documentation testcases 😉
budziq/rust-skeptic#37

mshal added a commit to mshal/cargo that referenced this issue Aug 3, 2017

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
Python data structures. The 'dependencies' structure lists the
dependencies between steps, and the 'commands' structure lists the
information required to run the commands from an external build system.

Fixes #3815
@mshal

This comment has been minimized.

Show comment
Hide comment
@mshal

mshal Aug 3, 2017

Contributor

I made an attempt at implementing this as a way of learning Rust. So far it just prints out the info from DependencyQueue and the rustc command-line invocations as Python data structures. With this I'm able to import the data structures into a Python program that writes out Tupfiles, and build a small dummy Rust program (a hello-world with two local library dependencies) with tup.

There are a number of things missing so far, so this isn't ready to land. In particular:

  1. build.rs support is not yet implemented. Confusingly, it will output a build step for a build.rs file with the same name as the rustc invocation.
  2. The final output of a program is not copied. It will create a rule for target/debug/deps/hello_world-b6a758bc870b5332, but not target/debug/hello_world, for example.
  3. Running cargo with --build-plan still creates some directories under target/. This should probably be left to the external build system.
Contributor

mshal commented Aug 3, 2017

I made an attempt at implementing this as a way of learning Rust. So far it just prints out the info from DependencyQueue and the rustc command-line invocations as Python data structures. With this I'm able to import the data structures into a Python program that writes out Tupfiles, and build a small dummy Rust program (a hello-world with two local library dependencies) with tup.

There are a number of things missing so far, so this isn't ready to land. In particular:

  1. build.rs support is not yet implemented. Confusingly, it will output a build step for a build.rs file with the same name as the rustc invocation.
  2. The final output of a program is not copied. It will create a rule for target/debug/deps/hello_world-b6a758bc870b5332, but not target/debug/hello_world, for example.
  3. Running cargo with --build-plan still creates some directories under target/. This should probably be left to the external build system.
@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Aug 3, 2017

Member

Just a quick note -- the Cargo team is actively hashing out high-level design questions around this and other aspects of build system integration. Hope to have some writeups soon!

Member

aturon commented Aug 3, 2017

Just a quick note -- the Cargo team is actively hashing out high-level design questions around this and other aspects of build system integration. Hope to have some writeups soon!

@Xanewok

This comment has been minimized.

Show comment
Hide comment
@Xanewok

Xanewok Aug 14, 2017

Member

@mshal awesome that you did some work on this!
If possible, I'd like to help with the implementation as it'd be very convenient to have that in the RLS, since right now we're going to use an approximation to perform our own builds using rustc.

Member

Xanewok commented Aug 14, 2017

@mshal awesome that you did some work on this!
If possible, I'd like to help with the implementation as it'd be very convenient to have that in the RLS, since right now we're going to use an approximation to perform our own builds using rustc.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Sep 2, 2017

Member

There's an RFC for build system integration now up at rust-lang/rfcs#2136

Member

alexcrichton commented Sep 2, 2017

There's an RFC for build system integration now up at rust-lang/rfcs#2136

mshal added a commit to mshal/cargo that referenced this issue Nov 17, 2017

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815
@mshal

This comment has been minimized.

Show comment
Hide comment
@mshal

mshal Nov 17, 2017

Contributor

Sorry for the delay - I've been distracted by other things. I have the latest attempt based off of cargo-0.22 here: https://github.com/mshal/cargo/tree/build-plan-0.22.0

I used 0.22 as the base because that's what we currently use in mozilla-central. I'm able to run cargo with --build-plan against the toolkit/library/rust/Cargo.toml file to generate a plan.json which contains (almost) all the information needed to invoke the rustc commands and build scripts with an external tool. The only missing thing I am aware of is the outputs of a build script, but those can be supplemented externally as a workaround.

It almost works when I run it through tup, but that has other problems with rustc's weird file usage patterns that still need to be resolved. I believe this is a reasonably complete implementation in the cargo side of things however.

I'd appreciate any feedback and improvements! If it would be easier, I could submit it as a pull request.

Contributor

mshal commented Nov 17, 2017

Sorry for the delay - I've been distracted by other things. I have the latest attempt based off of cargo-0.22 here: https://github.com/mshal/cargo/tree/build-plan-0.22.0

I used 0.22 as the base because that's what we currently use in mozilla-central. I'm able to run cargo with --build-plan against the toolkit/library/rust/Cargo.toml file to generate a plan.json which contains (almost) all the information needed to invoke the rustc commands and build scripts with an external tool. The only missing thing I am aware of is the outputs of a build script, but those can be supplemented externally as a workaround.

It almost works when I run it through tup, but that has other problems with rustc's weird file usage patterns that still need to be resolved. I believe this is a reasonably complete implementation in the cargo side of things however.

I'd appreciate any feedback and improvements! If it would be easier, I could submit it as a pull request.

@mshal

This comment has been minimized.

Show comment
Hide comment
@mshal

mshal Nov 17, 2017

Contributor

I forgot to mention, but I did convert the json file to a shell script that just invokes the rustc commands in the right order, and that shell script does complete successfully. So I think that shows it can at least function as a way of exporting commands to an external "build system"

Contributor

mshal commented Nov 17, 2017

I forgot to mention, but I did convert the json file to a shell script that just invokes the rustc commands in the right order, and that shell script does complete successfully. So I think that shows it can at least function as a way of exporting commands to an external "build system"

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Nov 20, 2017

Member

Awesome thanks for the update @mshal! A PR would probably be the easiest thing for review, yeah, so feel free!

Member

alexcrichton commented Nov 20, 2017

Awesome thanks for the update @mshal! A PR would probably be the easiest thing for review, yeah, so feel free!

@euclio euclio referenced this issue in steveklabnik/rustdoc Dec 14, 2017

Open

Not all required crates are added to doctests #219

mshal added a commit to mshal/cargo that referenced this issue Mar 27, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815

mshal added a commit to mshal/cargo that referenced this issue Apr 5, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815

mshal added a commit to mshal/cargo that referenced this issue Apr 23, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815

mshal added a commit to mshal/cargo that referenced this issue May 1, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815

mshal added a commit to mshal/cargo that referenced this issue May 3, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815

mshal added a commit to mshal/cargo that referenced this issue May 7, 2018

Add --build-plan for 'cargo build'
With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815

bors added a commit that referenced this issue May 10, 2018

Auto merge of #5301 - mshal:build-plan, r=matklad
Add --build-plan for 'cargo build'

With 'cargo build --build-plan', cargo does not actually run any
commands, but instead prints out what it would have done in the form of
a JSON data structure.

Fixes #3815

@bors bors closed this in #5301 May 10, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment