RFC for Public/Private Dependencies

[mitsuhiko]

This RFC introduces the concept of a public/private separation of crate dependencies to aid the ecosystem to explicitly deal with dependencies that themselves become part of the public API of a crate.

Rendered

[untitaker]

This appears to contradict the answer to the previous question. Is it or is it not allowed to do this? If not, doesn't this break backwards compat a bit harshly?

[untitaker]

s/their/there/

[untitaker]

s/extend/extent/

[untitaker]

s/dependencies/dependency/

[untitaker]

s/last/list/

[untitaker]

Are there concrete usecases where these annotations would help? Those would be worth mentioning here (primarily to pitch this RFC to unconvinced readers)

At least when I started writing Rust, I upgraded dependencies in a PATCH release without thinking about the implications for reexported APIs. This change could make crate authors think about this from the start. The last bulletpoint in "Unresolved Question" also hints at this.

[untitaker]

• You already hint at libcore and libstd, but to me this part is missing justification for why we should allow normal crate authors to disable this warning. An example where this becomes necessary might help.
• At https://github.com/rust-lang/rfcs/pull/1977/files#diff-396833e435181d84da75db4b1f53ef8dR137 you list a few cases where one might think it'd be fine to use this. For example when leaking impls of external traits. Which usecases do we encourage?

[mitsuhiko]

I rather not go into too much detail about this as this is a big unknown. I think it's largely irrelevant for this RFC anyways because we initially start out with just compiler warnings. Where to go from there will be seen by how much damage this does.

[untitaker]

Not a review, but random thought: You might have a public dependency that changes its major version but doesn't change the part of the API you reexport, or perhaps it just bumps its major version too aggressively.

[withoutboats]

This is still a breaking change in your library.

Suppose you and another crate foo both depend on the same version of bar, and both re-export bar::Bar. I depend on both you and foo, and pass a Bar I got from you to foo in my code. This works fine, because your Bar is the same type as foo's Bar.

bar releases a breaking change, which does nothing to the definition of Bar. You update to that major version, but don't do a breaking change because you believe nothing in your code has changed.

However, now its impossible for cargo to unify your dependency on bar with foo's dependency on bar, because they're different major versions. This means that you::Bar and foo::Bar are no longer the same type. If I upgrade my dependency on you, I will get a breakage, so you have made a breaking change.

[untitaker]

Good point, I completely overlooked that aspect. I have a few other concerns/questions about this feature (that don't really affect this PR), but let's cut this discussion short to not clutter this PR.

[mathstuf]

What if you relax the range on bar to include the new version rather than just bumping to only using the latest version. What kind of bump does that mandate?

ISTR seeing that it would be a major change due to cargo always choosing at the high end of the version constraint range. Would it be worth an RFC to cargo to have it see dependencies of >= 1.1, < 3 and ^1 in different crates to, instead of deciding on two copies, one of version 2.x and another of, say, 1.5, instead see that 1.5 satisfies both and ignore the 2.x version?

[untitaker]

@mathstuf yeah this is what I thought about as well but consider off-topic for this RFC

[untitaker]

This paragraph has confusing wording, at least to me. To reiterate the point of this example, you could say that "public" is a property of the dependency relation (the edge in the dependency graph) and not of the crate or the package (the node).

[mark-i-m]

Question: if your crate depends on multiple versions of the same dependency, does the compiled binary just have multiple copies of the same generated code? How does this affect name resolution in the compiler? Or linking? Also, is the binary proportionally larger?

[bbatha]

Public dependencies shouldn't just be restricted to public APIs. For instance, most -sys crates can only be used once due to C's linkage rules. Noting libc as a public dependency would have made may have limited the impact of the libc-ocolypse. It would also be convenient to have a notion of a "unique" crate for -sys crates and for #[allocator] crates but I think that feature is orthogonal, and complex enough that it should be left as a future extension.

[sfackler]

@bbatha libc does not have the single-version constraint. The libc-pocalypse pain was just the normal issue of crates trying to bridge libc 0.1 and 0.2 APIs.

There's already a notion of a "unique" -sys crate - setting the links field in Cargo.toml.

[joshtriplett]

If dependencies default to private, and you can't export types from private dependencies, then this introduces a compatibility break.

[joshtriplett]

(That said, I'd love to see a change like this.)

[mitsuhiko]

@joshtriplett that's why initially it will only be a compiler/cargo publish warning.

[withoutboats]

I think this is a real problem and this seems like a good solution, but it is a pretty big change. Even though you're starting with a warning, everyone is going to get the warning & be compelled to fix them. This is a huge burst of churn & the drawbacks section should at least note this as a drawback.

[untitaker]

s/bumb/bump/

[untitaker]

s/depenencies/dependencies/

[Ericson2314]

We also need to be able to express a private dependency which is locked with another dep's public dependency, see my running a -> c; a ->b; b -> c example from above.

[mitsuhiko]

@Ericson2314 is this really necessary? Why can you not just have a pin on the major version? eg: url = { version = "1", public = true }.

[Ericson2314]

@mitsuhiko actually it might not be, but we should make not of this in the semantics: just as your public deps must be unique version per name, so your private dependencies and their public dependencies must be unique version per name.

[eternaleye]

I think this is one place where @bbatha's concern (link-only-once) diverges from what's described by (and very useful for) this RFC, and the two probably need handled distinctly.

In particular, in the absence of link-only-once dependencies, a private dependency hides all transitive dependencies behind it - and this is hugely useful in some cases.

However, link-only-once crates must not be hidden in this manner.

As a result, I think these might be better served by orthogonal mechanisms.

[mitsuhiko]

Yeah. I am not entirely sure how to best deal with link only dependencies. They are definitely iffy and out of the scope as far as I'm concerned. I will add a section about those to the RFC.

[Ericson2314]

Agree that it's out of scope. Simply requiring that all link-once deps be reachable through chains of public dependencies is sound, but way overly restrictive.

[mathstuf]

How about cratename = { from_dependencies = true } (or similar)? You're relying on deps to not break API with their deps being bumped, but that's true anyways already.

[untitaker]

I don't understand why from_dependencies is a better name than public, or how the version could be inferred. Could you give an example?

[mathstuf]

It'd be a replacement for version = "…" and an indication that cargo should figure out the version based on public dependencies of dependencies instead of trying to give a constraint directly. Basically "I'm fine with whatever my dependencies need because that's how I use this crate".

[untitaker]

I see, as far as I understand this could be a different RFC independent from this one though.

[mitsuhiko]

I think people should just pin less strictly instead. It's fine to just pin for 1 for instance.

[mathstuf]

But if I use a crate only because dependencies use it, why not have a way to say "I use it only because dependencies make me use it; let them tell me what version I need"?

In any case, probably a topic for a separate RFC.

[mitsuhiko]

But if I use a crate only because dependencies use it

If you only use it because dependencies use it and it's not part of their APIs the dependency is invisible to you. If it's a visible (public) dependency then the version of the dependency is relevant to you.

[Ericson2314]

I don't really like this language. If you depend on items from a dependency (e.g. types) even if you don't reexport them, you still need a public dep. Also----and this is crucial---if you don't reexport anything, it's not a breaking change to change your public deps if you're own interface doesn't change.

Consider a situation where a depends (privately or publicly) on b which publicly depends on c. To use any items from b with interfaces relying on c, a should also need to depend on c (and constrain that dep to be unified with b's). But now the extra dep of a on c forces the "effective interface: of b with given c to not break. Conversely, if a doesn't do that, then we know b's use of c is irrelevant so the relevant parts of b also won't break.

This is an instance of the general principle that the version of a crate is like a fallback / catchall that describes the remnants of the interface not already accounted for in the crate metadata, and constrainable with a dependency. Public deps are accounted for and can be constrained downstream, and thus need not cause interface breakage in and of themselves.

This may sounds like needlessly fancy reasoning, but this is actually really important practically. Breaking changes already are difficult growing pain with large ecosystems, and would become impossibly so if on every upstream breaking change, all publicly depending downstream was forced to issue their own breaking change even if all they did is bump a dependency. Conversely, tiny breaking changes are far easier to deal with if every publicly depending downstream libraries that aren't affected (i.e. most) just need to make a new release with a relaxed upper bound on their public dep).

[mitsuhiko]

If you depend on items from a dependency (e.g. types) even if you don't reexport them, you still need a public dep.

Can you clarify what you mean by that? If you do not re-export them there is no need for a dependency to be public.

[Ericson2314]

Perhaps we are using "re-export" differently? To me, reexport means you include the *definition" in your interface, i.e. with a pub use, as opposely merely using the dependency's items. Reexporting is bad thing to do because then the public dep really does become part of your API in ways downstream may not control --- merely bumping a version bound over a breaking change may indeed be a breaking change.

[mitsuhiko]

Correct. If you accept a type from a crate as parameter it means it's re-exported in your API.

[Ericson2314]

Uh I am not sure which you are agreeing with?

[mitsuhiko]

Maybe I'm just to dumb to understand the comment but I do not quite follow what the difference here would be. Is the idea that if you use a subset of b that does not c you do not want a public dependency to c? That should be covered anyways.

[Ericson2314]

@mitsuhiko With the terminology change, the only remaining issue is the sentence: "Effectively the idea is that if your own library bumps a public dependency it means that it's a breaking change of your own crate."

I realize now this sentence is doesn't actually effect what this RFC specifies, so as far as the detailed design goes we're all good---feel free to just cut that sentance and ignore the rest of this post :).

But for the record, actually a changing of a dep should never be a breaking change. In the private dep case, downstream cannot tell at all, so we're good. In the public dep case, the solver will simply not use the new crate if the public dep cannot be unified with other public deps.

I wrote before

This is an instance of the general principle that the version of a crate is like a fallback / catchall that describes the remnants of the interface not already accounted for in the crate metadata, and constrainable with a dependency.

I think I have a better way of describing things. The general maxim for compat is "if I publish this crate, in all solutions where this this crate could be substituted for another, things must work in both case". If there is a solution where the substitution breaks, we could try to fix that case, or we could try to rule it out.

Since public deps may be exposed but also must exist once, any bounds adjustment is OK because the version unification would rule out laxer bounds switching to another version the other crates in the plan cannot cope with.

Mathematically, one can view a crate wrt compatability as Map<PubDeps, Map<Name, Item> (maps are partial functions). Just as adding definitions to a crate preserves compatibility, so relaxing bounds does too---both just extend domains of the partial functions. Whereas removing definitions does not preserves compatibility, tightening bounds is fine because solver-time failures are fine (the solver just moves on) whereas build-time errors aren't.

[Ericson2314]

Hmm @alexcrichton in #1977 (comment) you wrote

When libc reaches 1.0 then many crates will need to bump their major version as libc is a public dependency.

But per this thread it would only be a minor bump. Using this side-thread as it's not really a core part of the RFC but do want to get this clarified.

It's possible we'd need to tweak some things like method resolution for what I said to actually be true. (e.g. do the inherent methods of a reexported type "vagabound" with the type and are usuable via the export?)

[Ixrec]

I believe Alex meant all the crates with a public dependency (direct or transitive) on libc would need a major version bump when libc goes 1.0, but all the crates with a private dependency (direct or transitive) on libc would not. That's why the next sentence of that comment is:

This RFC will allow us to precisely identify what set of crates need to be bumped, transitively!

Where I believe "need to be bumped" refers only to major version bumps.

[Ericson2314]

Right, I'm trying to argue that, absent reexports (as opposed to mere exposing of departure) changing dependencies in any way is not a breaking change because every plan the build could go wrong the solver will disallow anyways.

[kornelski]

Is the crate using the dependency the right place to declare it?

It could be a declaration in the dependency itself, similar to the link attribute (i.e. the dependency would mark itself as intended to be shared and exist only once [within any subtree of dependencies?]).
IMHO that would be a significant improvement in usability, because it frees all users of the crate from understanding the concept and marking their dependencies appropriately.

Perhaps it could even be a declaration per type or module? (e.g. pub(globally unique) or pub(no re-export).) e.g. an Error type may be intended to be re-exported, but other types should not.

[kornelski]

Does it have to be explicit at all? Can cargo/rustc notice the situation and emit a warning? (and you could silence the warning with either allow attribute or some cargo.toml flag)

warning: you have two copies of foolib, because bar requires 1.x and baz requires 2.x

I'm assuming duplication of dependencies is generally unexpected (it's not a thing in C) and undesirable (it bloats executables), so users may want to avoid it even if it doesn't break any APIs.

[untitaker]

@pornel I think your proposal misses the point of this one. The point isn't to ensure that a dependency is used only once in the entire dependency graph, but to provide cargo with additional metadata that can then be used to potentially implement things like hinted at in the "Unresolved questions" section, or just improve the error messages that currently happen at version splits.

I'm assuming duplication of dependencies is generally unexpected

No, it's completely reasonable to have that for private dependencies.

You're right that this situation could potentially be detected by the compiler, however IMO this might turn out to be too complex (or slow for certain operations) to implement since now your dependency management (e.g. cargo update) depends on the sourcecode as well, not just the metadata.

[yodaldevoid]

Some related discussion: rust-lang/cargo#2064

[eternaleye]

I think you're missing something - specifically, I think this may require changes to the index format in order to allow Cargo to make dependency-resolution decisions based on whether dependencies are public.

Without changes to the index, either Cargo will still need to perform conservative "everything is public" resolution (and the situation does not actually improve), or it may perform "optimistic" resolution (as if everything is private) and decide on a resolution that it cannot tell is invalid until after it's downloaded the crate archives (which is wasteful, and offers no clean recovery path).

[mitsuhiko]

@eternaleye why would cargo have to assume a public dependency?

[Ericson2314]

@mitsuhiko this is about whether the solver guess solutions which it then checks against dependency privacy, or whether it has access to privacy while solving---the latter is probably far more efficient as partial solutions can be ruled out earlier.

[mitsuhiko]

@Ericson2314 cargo knows what dependencies are public from the definition in the Cargo.toml. The only thing the #[allow(...)] does is silence the warning (later error). I don't believe it needs to impact the dependency resolution algorithm but I might be missing something here.

[eternaleye]

@mitsuhiko: The problem here is with the privacy/publicity of transitive dependencies. Imagine the following dependency graph:

• A
  • B
    • D
    • E
  • C
    • E

All dependencies are public, and furthermore let us presume that these "E" dependencies are ones that, if one of them were private, would fall into resolutions that "previously [were] prevented from compiling by restrictions in cargo."

I clone A, and run cargo build. Cargo's dependency resolution does not have access to the Cargo.toml for anything other than A. For everything else, it accesses the index. As a result, in order to know whether B and C depend on E privately or publicly, the index must carry this information.

If it does not, Cargo has a few options:

1. Presume they are all public, and reject the resolution. No improvement over today.
2. Presume they are all public, and if the resolution would be rejected, fetch all potentially eligible crates in order to re-perform resolution using their Cargo.toml files. Incredibly network-intensive, may fail anyway, significant new logic.
3. Presume they are all private, begin fetching the selected crates (which may have version skews that are not permissible!), and then discover then that the resolution is irresolvable. Network-intensive, offers no usable error handling path where some versions of B (or C) make E private and others make E public.

[mitsuhiko]

@eternaleye in case you are referring to the crates.io index, yes that information (public true/false) would be contained in the "deps" section.

[mitsuhiko]

I for some reason thought you meant that #[allow(external_private_dependency)] needs to be reflected in the index.

[nikomatsakis]

👍 to the general idea. Some random comments:

• I think assuming private dependencies by default is an interesting choice: probably the right one, but not obviously so. I'd be curious to see some measurements on how often deps in practice are private vs public (I don't really know).
• I'd like to hear a lot more details about what rules the compiler will use when issuing a warning.
• This also seems related to the "private type in public API" rules (which have been a perennial point of controversy). I don't want to join the two topics together per se, but it seems to me that they are related in purpose. The existing rules aim to ensure that private structs are not exposed or usable outside your crate, and the same is true of the external deps.

[Ericson2314]

I don't want to join the two topics together per se, but it seems to me that they are related in purpose.

I'm afraid we might. But maybe this isn't so bad. Comparability guarantees in conjunction wit this can be a clear and clearly-motivated way (even if those are only "guarantees" with warnings-as-errors) to evaluate those rules.

[mitsuhiko]

@mark-i-m i missed the question but the RFC changes nothing (so far) about how multiple dependencies functionally work in Rust. You can already end up with multiple versions of the same crate for as long as major versions are involved.

[alexcrichton]

Thanks for writing up this RFC @mitsuhiko! Here's some thoughts I had while reading:

• I believe @eternaleye is correct where the fact that a dependency is either public or private needs to be encoded into a registry's index. This is sort of like how the "kind" of dependency (build, dev, etc) is encoded today. This is I believe a very easy change, but is likely worth mentioning.
• The RFC explicitly calls out how on a publish you'll get warnings, but this is happening on all rustc compilations, right? (that's what I'd like to happen, at least).
• I, like @nikomatsakis, wouldn't mind seeing some more detail on the compiler side of things. Both in terms of what the argument looks like that Cargo is passing but also the way the compiler is detecting these apis.
• The error messages here may run a risk of being confusing in some situations. For example let's say we've got A -> B -> C where B has a public dependency on C. This means that A can reexport some of C's types, but if you're a Cargo user then C could be way far down in the compilation graph and totally unknown. Do you think the compiler should go the extra mile and detect that C was exported transitively through B and tell A that B needs to be a public dependency? Put another way, if B is listed as a public dependency of A and then A reexported some of C's types, would the compiler warn/error about that?
• Currently there's no mention of crate resolution in Cargo, but I find that to be a crucial aspect of public/private dependencies. Cargo will specifically at least need to start rejecting crate graph resolutions where two versions of a crate are publicly reachable to one another. This in turn would fix almost all of the woes brought up during Cargo should understand distinctions between public/private dependencies cargo#2064 because Cargo will deterministically reach a statisfiable and compilable graph.

I think it's also worth mentioning in general some use cases of this RFC to mitigate pain seen in the ecosystem today. Note that this is mostly just an example, I'd encourage discussion of precisely what's happening here to happen elsewhere so this thread can continue to focus solely on this RFC. I consider public/private dependencies to be a big part of the answer to the major version upgrade of dependencies like libc. Once this change is implemented and propagated, I would feel confident about bumping the libc crate to 1.0.

When libc reaches 1.0 then many crates will need to bump their major version as libc is a public dependency. This RFC will allow us to precisely identify what set of crates need to be bumped, transitively! So immediately we'll have precise knowledge of how much of the ecosystem needs to be upgraded. After that Cargo will also reject any attempt to compile a graph which contains both libc 0.2 and libc 1.0 which would fail to compile. This is the crucial part of crate resolution I mentioned in the last bullet above. If Cargo determines that types from libc 0.2 can reach libc 1.0 then it will reject the resolution before we even start running the compiler. This means that error messages are immediate, high quality, and actionable.

Overall public/private dependencies empowers crate authors to release new major versions and:

• Precisely know the impact of how many crates need to have a major version bump
• Prevent any weird compiler error messages by having two versions of the library reach each other
• Allow consumers to know exactly what crates need to be updated so they know where to file bugs and/or focus efforts

Note that public/private dependencies does not reduce the amount of work that needs to be done as part of a major version upgrade, it simply makes the experience of discovering the work before/after the major bump realistic. Similarly it enables authors to have precise knowledge of when they're done upgrading, because it's when Cargo starts to invoke the compiler!

[cuviper]

Please remember to keep an eye on compatibility, especially if there are to be changes to the crates.io index.

[Ericson2314]

@alexcrichton

• The error messages here may run a risk of being confusing in some situations....

I'm very interested in this bullet, but having trouble following it.

---

edit I think I see all the points now.

In general I'd like to come up with a plan that prevents all such possible broken builds we care about, and then scale it back to something implementable. This would be opposed to figuring out something that seems feasible to do and then figuring out what build failures it prevents.

This stuff is already is quite hard in a simple language....and then throw in traits and my mind melts.

[SimonSapin]

This basically means that if you privately depend on Hyper 0.3 and Hyper 0.4, that's an error.

Does that mean that it is impossible for a given type to implement serde::Serialize from both serde 0.9.x and serde 1.x.y? (Either through reexports like https://github.com/serde-rs/legacy, or with a future Cargo feature to locally rename dependent crates.)

That sounds bad :/

[mathstuf]

Maybe if you have to go through some kind of "legacy" crate meant for bridging support for multiple versions of a single crate? There could be a field for that.

Though you still would have the problem that serde_derive can only generate for a single version of serde at a time…

[nox]

Why distinguish public and private dependencies if it means you can't depend on different versions of a private dependency?

[yodaldevoid]

This basically means that if you privately depend on Hyper 0.3 and Hyper 0.4, that's an error.

This feels like an error or a case of a bad explanation. If this is true as I am reading this it would be a major change in behavior when it comes to crate version resolution that I see breaking many crates. Can we get some clarification on this?

[withoutboats]

@alexcrichton's comment I think needs more context. What he meant was that if two versions of hyper are "exposed" to you, its an error. A dependency is exposed to you if:

• It is a direct dependency of your crate, public or private.
• It is a public dependency of a direct dependency or another public dependency of a direct or public dependency (recursively into infinite).

That is, this error is exactly what the RFC is supposed to introduce. It won't break any existing code, because dependencies are marked private by default and you can't directly depend on the same crate twice (at least not in a way that would trigger this error).

Obviously, there are use cases for "version straddling" - like implementing Serialize from both versions of serde. This needs to be supported somehow, possibly involving an opt out of this error. If its not adequately resolved in the RFC text I would say we make it an unresolved question for now.

[shepmaster]

if two versions of hyper are "exposed" to you, its an error

In @SimonSapin's examples

reexports like https://github.com/serde-rs/legacy

Legacy would treat serde 0.9 as a private dependency, so all's well here.

a future Cargo feature to locally rename dependent crates

Would depend on how it's implemented, of course, but I imagine that it could work in a similar way. The important thing would be that the renamed crate couldn't have any public dependencies of its own that then conflict. For example, if crate A v1.0 has a public dependency on Common v1.0 and A v2.0 has a public dependency on Common v2.0, then I'd hope the warning / error would be raised (unless there's something that also renames the public dependencies...)

[SimonSapin]

Legacy would treat serde 0.9 as a private dependency, so all's well here.

So implementing serde09::Serialize for a public type does not make serde09 a public dependency?

[shepmaster]

So implementing serde09::Serialize for a public type does not make serde09 a public dependency?

I believe it would make the crate serde09 (version 0.9) a public dependency, but it would not make the crate serde (version 0.9) a public dependency.

This assumes that the crate performing the implementation of serde09::Serialize has chosen to mark serde09 as a public dependency (as the default is to act as a private dependency).

[withoutboats]

The real question is whether or not its sufficient for serde09 to just not marke serde a public dependency, or if we need some other mechanism here. I'm really unsure!

[rfcbot]

The final comment period is now complete.

[withoutboats]

Merged this PR! Tracking issue is rust-lang/rust#44663. Thanks so much @mitsuhiko, @carols10cents, and everyone else for working on this RFC.

Excited to try to get this in nightly during the upcoming impl period. 🎉

[alekratz]

The rendered RFC link gives a 404 - could it be updated to the current repository's copy please?

[carols10cents]

@alekratz

The rendered RFC link gives a 404 - could it be updated to the current repository's copy please?

Done!