inner crates, aka multiple crates per file

[pnkfelix]

Proposal

Summary and problem statement

Add ability to express multiple crates (with corresponding multiple output object files) in a single source file.

Syntax might look something like:

crate first {  // generates `first.rlib`
  pub static DEF: i32 = 3;
}

crate second { // generates `second.rlib`
    extern crate first; pub use first::DEF as D;
}

// we will compile above to completion before
// we process body of (implicit) crate for source file.

extern crate second;

fn main() { println!("{}", second::D); } // prints "3"

Motivation, use-cases, and solution sketches

The problem: Today, any Rust example involving more than one crate or more than one code-gen unit requires multiple source files to demonstrate. This means you cannot use play.rust-lang.org for such cases. It means that people seeking to describe multi-crate scenarios in documentation or blog posts have to describe multiple files, or rely on the reader to infer such structure. It means that people seeking to describe multi-code-gen unit problems also end up having to either use multiple crates (and thus hit the problems above), or they rely on details of how a source file is mapped into separate code-gen units by the rust compiler.

One very important case that requires multiple crates today: procedural macros. I.e., you are forced to define a procedural macro in a separate crate from where it is used.

By adding the ability to define extra crates inline, we will side-step all of the above problems.

(Its entirely possible that this feature will be of most use to compiler developers, but I have tried to identify cases, like documentation and proc-macros, that have a more general audience. No matter how you slice it, its a niche group.)

Update: I am explicitly removing proc-macros from the scope of this proposal, due to it raising thorny issues (discussed on zulip) that I was not intending to try to address as part of this proposal.

Links and related work

Few languages support this (which to be honest I'm definitely thinking could be a sign that its a linguistic anti-pattern). The closest thread I can see is work on "multi-stage programming", but even that connection seems like a bit of a stretch.

Initial people involved

• Owner, if known: pnkfelix wants to drive this
• Liaison

What happens now?

This issue is part of the lang-team initiative process. Once this issue is filed, a Zulip topic will be opened for discussion, and the lang-team will review open proposals in its weekly triage meetings. You should receive feedback within a week or two.

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[rustbot]

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

[jhpratt]

Hey @pnkfelix, I've thought about this some on my own for a while now; I'm just now seeing this issue. There's definitely different approaches that can be taken here.

My thought was to expand on the idea of workspaces, having the ability to expose one "primary" crate (the others being internal). This would mean that the compiler could split each internal crate into its own codegen unit, but visibility and coherence would (optionally) apply on the workspace level. The details of this would be entirely hidden to the end user, as they'd just add a single crate to Cargo.toml.

I believe that a change like this would require significantly fewer changes to the compiler, as there's no new syntax and crate resolution would work exactly the same as they do now.

I will acknowledge that this does not in any way work to solve the motivation of the playground, blog posts, etc. One additional motivation: hiding crates that are implementation details of another crate. For example, the time-macros crate is only ever intended to be used to time.

[pnkfelix]

I'm not seeing sufficient enthusiasm for this from the team to justify keeping this proposal open. (I had to scale back its ambitions after I realized that the proc-macro case was not handled "out of the box".) So I am closing it.

I still think the overall concept has merit, but maybe its better implemented as prepass solely within cargo or something along those lines.