eRFC: Cargo build system integration

[aturon]

This experimental RFC lays out a high-level plan for improving Cargo's ability to integrate with other build systems and environments. As an experimental RFC, it opens the door to landing unstable features in Cargo to try out ideas, but not to stabilizing those features, which will require follow-up RFCs. It proposes a variety of features which, in total, permit a wide spectrum of integration cases -- from customizing a single aspect of Cargo to letting an external build system run almost the entire show.

Rendered

[aturon]

cc @rust-lang/cargo @jsgf @sid0 @acmcarther @luser @rillian

[cramertj]

homogeneous

[est31]

Can be written both ways apparently https://en.oxforddictionaries.com/definition/us/homogeneous

[chris-morgan]

That citation indicates that homogeneous is still more popular (“and around a third of citations for the word now use the form homogenous”). Homogenous was an error that has only gained any legitimacy because so many people make it! (As I like to say: language is a popularity contest.)

[cramertj]

homogeneous

[cramertj]

homogeneous etc...

[cramertj]

This and the one below should be [pkgconfig] so that the link behaves :)

[sunshowers]

Thanks for writing this! Here's some really early feedback after a quick read.

[sunshowers]

Thanks for writing this! Here's some really early feedback after a quick read.

[sunshowers]

By homogeneous build systems I assume you mean Bazel, Buck etc. Is this particular case something that an organization has expressed interest in? I'm not really sure how this is going to work with the build hermeticity/reproducibility constraints that Buck has.

For context, at Facebook we use the same system to manage dependencies from crates.io as we do to manage external C, C++ or Python dependencies. So our current workflow fits the "unmanaged crates" case somewhat better.

[softprops]

This is somewhat the story for bazel as well.

It's very noticeable as well when working other typically managed dependency languages that these build systems really want to you to vendor your dependencies. Cargo like mvn or ivy for the jvm tries to push you in the opposite direction with remote but managed dependencies.

I don't really think one is better than the other but they do conflict in assumptions and philosophy.

[sunshowers]

Would it be possible to invoke the native build system directly instead of using cargo build?

---

I'm not sure if this is too low-level for this RFC, but it might be worth talking a bit about where build artifacts go. With Buck, build artifacts always go into buck-out in the root of the monorepo. This is advantageous for a few reasons:

• it avoids polluting random subdirectories
• it saves on Watchman file monitoring resources

Would tools like the RLS know to read build artifacts from the buck-out directory instead of from target/rls?

[sunshowers]

I think this is my biggest concern here. I don't think changes in the way native deps are built currently cause a semver bump. This change would seemingly propel build.rs into an API, so should any substantive changes here cause a semver bump?

If build.rs builds the native dependency with a set of features today (./configure --with-something), will it be the job of the native build system to ensure that its own copy of that dependency has those features turned on?

[sunshowers]

Hmm, I'm not sure the later section really addresses how homogeneous build systems will tend to use native dependencies. That section seems to be restricted to how packages on crates.io specify native dependencies. That's an important part of the story, of course. But ideally, authors of Rust crates within Facebook's monorepo would not get Cargo involved in their native dependencies at all. They would use standard Buck rules to specify their native dependencies, and Buck would build those deps and provide libraries that Rust could link against.

[mark-i-m]

Hmm... That's very similar to what I've been doing for OS projects to link the kernel with external assembly code, except the other way around. I have found it easier to compile my rust code into a .a, compile my other stuff into .os and pass them all to ld.

The annoying part of this approach is that if you want to use any other rust libraries (e.g. liballoc or libcollections) you also need to obtain them and pass them to either rustc or ld... It gets to be a bit messy if your not careful.

[aturon]

I've updated this section to try to make things more clear. The setup is really very simple: Cargo/rustc don't need to know much at all about these native deps, other than their location. The build system manages the rest.

[sunshowers]

Any chance you could avoid the use of first-person pronouns here? "We" and "our" are a little confusing.

"it's where Cargo's model of semver comes into play" etc seem clearer.

[sunshowers]

What sort of timeline are you looking at for "immediately pursu[ing] extensibility points"?

[sunshowers]

This could be turned into a column of the table for clarity. "Scope" -> "Entire dependency graph"/"Individual crate"

[softprops]

Typo here. Proeject should be project

[est31]

a their

[aturon]

Oy, also cc @joshtriplett of course!

[aturon]

@sid0 I've just pushed significant clarifications for the homogeneous case and for native dependencies. Please let me know if the result aligns with your understanding.

[jpakkane]

A while ago I wrote a bit on dependencies and related things in general.

[joshtriplett]

As written, this looks very good to me. In particular, I like that you've provided a detailed look at the problem, and the collected discussions and information we've obtained, while holding off on providing concrete solutions.

There are a couple of aspects I'd appreciate some revisions on, in being both a little less specific about solutions and a little more specific about problems.

I'm not quite comfortable yet with us emphasizing "build plans" or "Cargo plugins" as possible solutions. I do think it's important that we emphasize the idea of letting Cargo provide "builds as a library" that can be woven together as part of a larger build system, but I'd really like to see that explained in a way that takes the "midlayer mistake" into account, and especially, I think describing it as "Cargo plugins" implies "fit your plugin into Cargo" rather than "use Cargo's steps as a library and override some of them". Would you consider generalizing this a little bit, and discussing it a bit more, focusing on the problem and the properties of a correct solution, rather than any specific solution? (Please feel free to take some text from the last paragraph of my pre-RFC if that helps.)

This paragraph from the eRFC seems very promising to me:

Note that the four-step division above is a useful conceptual model, but is ultimately too coarse-grained; we will likely need to divide up Cargo's functionality into numerous small pieces that can be re-used when integrating into a larger build system. This finer division is left as a question for experimentation.

I'd just like to see other parts of the eRFC match that more in spirit.

On the flip side, there are also certain critical properties we'd like a Cargo-based build system to provide, and there's an important aspect here that Cargo (and the crates.io ecosystem) exists to provide policy, not just mechanism. I think it's worth emphasizing, in the eRFC, that Cargo is intentionally a somewhat opinionated build system, precisely because it's solving a very difficult problem that's looks easier than it is, and hard to get right in all the details. And part of the goal here is to provide the flexibility to integrate Cargo with other build systems, distributions, and external dependencies, while continuing to provide policy as appropriate, not abandoning that and just serving up mechanism alone.

Do those points seem like reasonable additions, to you?

[vorner]

I wonder if using rust completely without cargo is even considered as possibility. Here, the RFC doesn't seem to think so.

I guess cargo does a lot of useful work, but maybe part of that work is somewhat „artificial“. If I build a C application, I build just my sources, the build system to compile each .c file into .o file is reasonably straightforward. If have a dependency, I don't build it, but expect it to already live in my system. This approach doesn't seem to be possible right now (eg. a lot of code I write depends on serde… it would make sense to have already compiled serde somewhere in the system, as a dynamic library or something and reuse).

Anyway, that's just kind of brainstorming and I'm not sure this is completely related to the point of the RFC, so feel free to ignore. In general, I'm in favour of this RFC ‒ it at least states the problems and some ideas, even though I'm not 100% sure what the next actionable steps would be after accepting it.

[withoutboats]

@vorner this RFC is specifically a cargo team RFC to make it easier to integrate cargo into larger build systems. Invoking rustc without cargo is out of scope for this RFC.

[Ixrec]

nit: "employ a their own strategy"

[matklad]

@aturon for other two concerns, which boil down to "it's not obvious that people would want to use Cargo for internal crates at all". I think we should explicitly mention "imposing less control" for internal crates in homogeneous build system (that is, remove Cargo from equation completely), and explain why this route won't be able to fulfill our goals.

[aturon]

@matklad I've pushed an update that, I think, should address your concerns. In particular, it commits the Cargo team to working with the Dev Tools team to fully enumerate the needs of tools, and to explore the lightest weight way of providing that information (which may just be something that a build tool generates, rather than something that Cargo needs to mediate).

[matklad]

Thanks @aturon !

@rfcbot resolved cargo_for_internal_crates

[matklad]

@rfcbot resolved bazel_workflows

[matklad]

@rfcbot reviewd

[matklad]

@rfcbot reviewed

:)

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[joshtriplett]

@aturon

rfcbot doesn't know that @joshtriplett has joined the Cargo team, but we'll block on his review as well :-)

Already discussed in the Cargo call, but:

• @joshtriplett

👍

[sunshowers]

I reread this with all your changes, and I have nothing substantive to add. One change I really like is the possibility to just use an interchange format. I think that has the potential to, if not eliminate plugins completely, at least substantially simplify them.

Thanks!

[aturon]

I've started up a thread to try to get a firmer grasp on the needs of Rust tools. Please join in!

[rfcbot]

The final comment period is now complete.

[aturon]

Ooops, this somehow didn't get merged before. Doing so now!

I didn't open a tracking issue on the Rust repo, since this is for Cargo. I'm not sure yet how we'll want to organize/track work there, but I'll try to post back on this thread once there's something to look at.

[jjpe]

Could someone update the rendered link in the OP to a working URL? Unfortunately it's broken ATM...

[carols10cents]

Updated the rendered link to link to rust-lang/rfcs instead of aturon/rfcs. I wish this was automated!

[matklad]

cc rust-lang/cargo#5332