New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eRFC: Cargo build system integration #2136

Merged
merged 12 commits into from Feb 1, 2018

Conversation

@aturon
Member

aturon commented Sep 1, 2017

This experimental RFC lays out a high-level plan for improving Cargo's ability to integrate with other build systems and environments. As an experimental RFC, it opens the door to landing unstable features in Cargo to try out ideas, but not to stabilizing those features, which will require follow-up RFCs. It proposes a variety of features which, in total, permit a wide spectrum of integration cases -- from customizing a single aspect of Cargo to letting an external build system run almost the entire show.

Rendered

@aturon aturon added the T-cargo label Sep 1, 2017

@aturon aturon self-assigned this Sep 1, 2017

@aturon

This comment has been minimized.

Show comment
Hide comment
Member

aturon commented Sep 1, 2017

@aturon aturon changed the title from RFC: Cargo build system integration to eRFC: Cargo build system integration Sep 1, 2017

@sunshowers

Thanks for writing this! Here's some really early feedback after a quick read.

Show outdated Hide outdated text/0000-build-systems.md
Show outdated Hide outdated text/0000-build-systems.md
Show outdated Hide outdated text/0000-build-systems.md
Show outdated Hide outdated text/0000-build-systems.md
Show outdated Hide outdated text/0000-build-systems.md
Show outdated Hide outdated text/0000-build-systems.md
Show outdated Hide outdated text/0000-build-systems.md
@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Sep 2, 2017

Member

Oy, also cc @joshtriplett of course!

Member

aturon commented Sep 2, 2017

Oy, also cc @joshtriplett of course!

@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Sep 2, 2017

Member

@sid0 I've just pushed significant clarifications for the homogeneous case and for native dependencies. Please let me know if the result aligns with your understanding.

Member

aturon commented Sep 2, 2017

@sid0 I've just pushed significant clarifications for the homogeneous case and for native dependencies. Please let me know if the result aligns with your understanding.

@jpakkane

This comment has been minimized.

Show comment
Hide comment
@jpakkane

jpakkane commented Sep 2, 2017

A while ago I wrote a bit on dependencies and related things in general.

@joshtriplett

This comment has been minimized.

Show comment
Hide comment
@joshtriplett

joshtriplett Sep 2, 2017

Member

As written, this looks very good to me. In particular, I like that you've provided a detailed look at the problem, and the collected discussions and information we've obtained, while holding off on providing concrete solutions.

There are a couple of aspects I'd appreciate some revisions on, in being both a little less specific about solutions and a little more specific about problems.

I'm not quite comfortable yet with us emphasizing "build plans" or "Cargo plugins" as possible solutions. I do think it's important that we emphasize the idea of letting Cargo provide "builds as a library" that can be woven together as part of a larger build system, but I'd really like to see that explained in a way that takes the "midlayer mistake" into account, and especially, I think describing it as "Cargo plugins" implies "fit your plugin into Cargo" rather than "use Cargo's steps as a library and override some of them". Would you consider generalizing this a little bit, and discussing it a bit more, focusing on the problem and the properties of a correct solution, rather than any specific solution? (Please feel free to take some text from the last paragraph of my pre-RFC if that helps.)

This paragraph from the eRFC seems very promising to me:

Note that the four-step division above is a useful conceptual model, but is ultimately too coarse-grained; we will likely need to divide up Cargo's functionality into numerous small pieces that can be re-used when integrating into a larger build system. This finer division is left as a question for experimentation.

I'd just like to see other parts of the eRFC match that more in spirit.

On the flip side, there are also certain critical properties we'd like a Cargo-based build system to provide, and there's an important aspect here that Cargo (and the crates.io ecosystem) exists to provide policy, not just mechanism. I think it's worth emphasizing, in the eRFC, that Cargo is intentionally a somewhat opinionated build system, precisely because it's solving a very difficult problem that's looks easier than it is, and hard to get right in all the details. And part of the goal here is to provide the flexibility to integrate Cargo with other build systems, distributions, and external dependencies, while continuing to provide policy as appropriate, not abandoning that and just serving up mechanism alone.

Do those points seem like reasonable additions, to you?

Member

joshtriplett commented Sep 2, 2017

As written, this looks very good to me. In particular, I like that you've provided a detailed look at the problem, and the collected discussions and information we've obtained, while holding off on providing concrete solutions.

There are a couple of aspects I'd appreciate some revisions on, in being both a little less specific about solutions and a little more specific about problems.

I'm not quite comfortable yet with us emphasizing "build plans" or "Cargo plugins" as possible solutions. I do think it's important that we emphasize the idea of letting Cargo provide "builds as a library" that can be woven together as part of a larger build system, but I'd really like to see that explained in a way that takes the "midlayer mistake" into account, and especially, I think describing it as "Cargo plugins" implies "fit your plugin into Cargo" rather than "use Cargo's steps as a library and override some of them". Would you consider generalizing this a little bit, and discussing it a bit more, focusing on the problem and the properties of a correct solution, rather than any specific solution? (Please feel free to take some text from the last paragraph of my pre-RFC if that helps.)

This paragraph from the eRFC seems very promising to me:

Note that the four-step division above is a useful conceptual model, but is ultimately too coarse-grained; we will likely need to divide up Cargo's functionality into numerous small pieces that can be re-used when integrating into a larger build system. This finer division is left as a question for experimentation.

I'd just like to see other parts of the eRFC match that more in spirit.

On the flip side, there are also certain critical properties we'd like a Cargo-based build system to provide, and there's an important aspect here that Cargo (and the crates.io ecosystem) exists to provide policy, not just mechanism. I think it's worth emphasizing, in the eRFC, that Cargo is intentionally a somewhat opinionated build system, precisely because it's solving a very difficult problem that's looks easier than it is, and hard to get right in all the details. And part of the goal here is to provide the flexibility to integrate Cargo with other build systems, distributions, and external dependencies, while continuing to provide policy as appropriate, not abandoning that and just serving up mechanism alone.

Do those points seem like reasonable additions, to you?

@vorner

This comment has been minimized.

Show comment
Hide comment
@vorner

vorner Sep 2, 2017

I wonder if using rust completely without cargo is even considered as possibility. Here, the RFC doesn't seem to think so.

I guess cargo does a lot of useful work, but maybe part of that work is somewhat „artificial“. If I build a C application, I build just my sources, the build system to compile each .c file into .o file is reasonably straightforward. If have a dependency, I don't build it, but expect it to already live in my system. This approach doesn't seem to be possible right now (eg. a lot of code I write depends on serde… it would make sense to have already compiled serde somewhere in the system, as a dynamic library or something and reuse).

Anyway, that's just kind of brainstorming and I'm not sure this is completely related to the point of the RFC, so feel free to ignore. In general, I'm in favour of this RFC ‒ it at least states the problems and some ideas, even though I'm not 100% sure what the next actionable steps would be after accepting it.

vorner commented Sep 2, 2017

I wonder if using rust completely without cargo is even considered as possibility. Here, the RFC doesn't seem to think so.

I guess cargo does a lot of useful work, but maybe part of that work is somewhat „artificial“. If I build a C application, I build just my sources, the build system to compile each .c file into .o file is reasonably straightforward. If have a dependency, I don't build it, but expect it to already live in my system. This approach doesn't seem to be possible right now (eg. a lot of code I write depends on serde… it would make sense to have already compiled serde somewhere in the system, as a dynamic library or something and reuse).

Anyway, that's just kind of brainstorming and I'm not sure this is completely related to the point of the RFC, so feel free to ignore. In general, I'm in favour of this RFC ‒ it at least states the problems and some ideas, even though I'm not 100% sure what the next actionable steps would be after accepting it.

@withoutboats

This comment has been minimized.

Show comment
Hide comment
@withoutboats

withoutboats Sep 2, 2017

Contributor

@vorner this RFC is specifically a cargo team RFC to make it easier to integrate cargo into larger build systems. Invoking rustc without cargo is out of scope for this RFC.

Contributor

withoutboats commented Sep 2, 2017

@vorner this RFC is specifically a cargo team RFC to make it easier to integrate cargo into larger build systems. Invoking rustc without cargo is out of scope for this RFC.

@matklad

This comment has been minimized.

Show comment
Hide comment
@matklad

matklad Sep 16, 2017

Member

@aturon for other two concerns, which boil down to "it's not obvious that people would want to use Cargo for internal crates at all". I think we should explicitly mention "imposing less control" for internal crates in homogeneous build system (that is, remove Cargo from equation completely), and explain why this route won't be able to fulfill our goals.

Member

matklad commented Sep 16, 2017

@aturon for other two concerns, which boil down to "it's not obvious that people would want to use Cargo for internal crates at all". I think we should explicitly mention "imposing less control" for internal crates in homogeneous build system (that is, remove Cargo from equation completely), and explain why this route won't be able to fulfill our goals.

@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Sep 19, 2017

Member

@matklad I've pushed an update that, I think, should address your concerns. In particular, it commits the Cargo team to working with the Dev Tools team to fully enumerate the needs of tools, and to explore the lightest weight way of providing that information (which may just be something that a build tool generates, rather than something that Cargo needs to mediate).

Member

aturon commented Sep 19, 2017

@matklad I've pushed an update that, I think, should address your concerns. In particular, it commits the Cargo team to working with the Dev Tools team to fully enumerate the needs of tools, and to explore the lightest weight way of providing that information (which may just be something that a build tool generates, rather than something that Cargo needs to mediate).

@matklad

This comment has been minimized.

Show comment
Hide comment
@matklad

matklad Sep 20, 2017

Member

Thanks @aturon !

@rfcbot resolved cargo_for_internal_crates

Member

matklad commented Sep 20, 2017

Thanks @aturon !

@rfcbot resolved cargo_for_internal_crates

@matklad

This comment has been minimized.

Show comment
Hide comment
@matklad

matklad Sep 20, 2017

Member

@rfcbot resolved bazel_workflows

Member

matklad commented Sep 20, 2017

@rfcbot resolved bazel_workflows

@matklad

This comment has been minimized.

Show comment
Hide comment
@matklad

matklad Sep 20, 2017

Member

@rfcbot reviewd

Member

matklad commented Sep 20, 2017

@rfcbot reviewd

@matklad

This comment has been minimized.

Show comment
Hide comment
@matklad

matklad Sep 20, 2017

Member

@rfcbot reviewed

:)

Member

matklad commented Sep 20, 2017

@rfcbot reviewed

:)

@rfcbot

This comment has been minimized.

Show comment
Hide comment
@rfcbot

rfcbot Sep 20, 2017

🔔 This is now entering its final comment period, as per the review above. 🔔

rfcbot commented Sep 20, 2017

🔔 This is now entering its final comment period, as per the review above. 🔔

@joshtriplett

This comment has been minimized.

Show comment
Hide comment
@joshtriplett

joshtriplett Sep 22, 2017

Member

@aturon

rfcbot doesn't know that @joshtriplett has joined the Cargo team, but we'll block on his review as well :-)

Already discussed in the Cargo call, but:

👍

Member

joshtriplett commented Sep 22, 2017

@aturon

rfcbot doesn't know that @joshtriplett has joined the Cargo team, but we'll block on his review as well :-)

Already discussed in the Cargo call, but:

👍

@sunshowers

This comment has been minimized.

Show comment
Hide comment
@sunshowers

sunshowers Sep 22, 2017

I reread this with all your changes, and I have nothing substantive to add. One change I really like is the possibility to just use an interchange format. I think that has the potential to, if not eliminate plugins completely, at least substantially simplify them.

Thanks!

sunshowers commented Sep 22, 2017

I reread this with all your changes, and I have nothing substantive to add. One change I really like is the possibility to just use an interchange format. I think that has the potential to, if not eliminate plugins completely, at least substantially simplify them.

Thanks!

@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Sep 26, 2017

Member

I've started up a thread to try to get a firmer grasp on the needs of Rust tools. Please join in!

Member

aturon commented Sep 26, 2017

I've started up a thread to try to get a firmer grasp on the needs of Rust tools. Please join in!

@rfcbot

This comment has been minimized.

Show comment
Hide comment
@rfcbot

rfcbot Sep 30, 2017

The final comment period is now complete.

rfcbot commented Sep 30, 2017

The final comment period is now complete.

@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Feb 1, 2018

Member

Ooops, this somehow didn't get merged before. Doing so now!

I didn't open a tracking issue on the Rust repo, since this is for Cargo. I'm not sure yet how we'll want to organize/track work there, but I'll try to post back on this thread once there's something to look at.

Member

aturon commented Feb 1, 2018

Ooops, this somehow didn't get merged before. Doing so now!

I didn't open a tracking issue on the Rust repo, since this is for Cargo. I'm not sure yet how we'll want to organize/track work there, but I'll try to post back on this thread once there's something to look at.

@aturon aturon merged commit 96dce82 into rust-lang:master Feb 1, 2018

@jjpe

This comment has been minimized.

Show comment
Hide comment
@jjpe

jjpe Feb 7, 2018

Could someone update the rendered link in the OP to a working URL? Unfortunately it's broken ATM...

jjpe commented Feb 7, 2018

Could someone update the rendered link in the OP to a working URL? Unfortunately it's broken ATM...

@carols10cents

This comment has been minimized.

Show comment
Hide comment
@carols10cents

carols10cents Feb 7, 2018

Member

Updated the rendered link to link to rust-lang/rfcs instead of aturon/rfcs. I wish this was automated!

Member

carols10cents commented Feb 7, 2018

Updated the rendered link to link to rust-lang/rfcs instead of aturon/rfcs. I wish this was automated!

@matklad

This comment has been minimized.

Show comment
Hide comment
@matklad
Member

matklad commented Apr 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment