Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for RFC 1977: public & private dependencies #44663

Open
5 tasks
withoutboats opened this issue Sep 17, 2017 · 37 comments
Open
5 tasks

Tracking issue for RFC 1977: public & private dependencies #44663

withoutboats opened this issue Sep 17, 2017 · 37 comments

Comments

@withoutboats
Copy link
Contributor

@withoutboats withoutboats commented Sep 17, 2017

This is the tracking issue for rust-lang/rfcs#1977 - public and private dependencies in cargo.

Blocking stabilization:

@Boscop
Copy link

@Boscop Boscop commented Oct 27, 2017

Maybe it's useful to look at how this problem is approached in Haskell.
Here are some relevant slides: https://wiki.haskell.org/wikiupload/b/b4/HIW2011-Talk-Loeh.pdf

Here is an example of a subtle issue that occurs with private dependencies:

Btw, I found it through this post: http://harry.garrood.me/blog/purescript-why-bower/

@Eh2406
Copy link
Contributor

@Eh2406 Eh2406 commented Mar 24, 2018

Would Rust2018 be a good opportunity to make the braking changes to Cargo.toml?

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Mar 25, 2018

Maybe. Though such changes should be designed and proposed soon, have an easy migration path, and preferably only affect a minority of existing crates.

@Nemo157
Copy link
Member

@Nemo157 Nemo157 commented Nov 29, 2018

Thought I just had related to ecosystems like futures that provide a "core" stable crate along with less stable utilities re-exported from a "facade": items from a public dependency re-exported through a private dependency should be treated as public. (This is likely how the implementation would behave anyway, but having a testcase for this would be nice). Basically how I see using a tightly coupled ecosystem like that work is that you would declare both the "facade" crate and "core" crate as dependencies, but only the "core" crate as public:

[dependencies]
futures = { version = "10.17", public = false }
futures-core = { version = "1.2", public = true }

But then, in code you would never import items from the "core" crate, instead you would simply have public APIs that return types which have been re-exported through the "facade" from the "core" crate and rely on the compiler to warn you if you accidentally use something that didn't originate in the "core" crate.

@Eh2406
Copy link
Contributor

@Eh2406 Eh2406 commented Nov 29, 2018

I don't quite understand something about your example. Can you elaborate on what the Toml and the code in the futures crate look like?

@Nemo157
Copy link
Member

@Nemo157 Nemo157 commented Nov 30, 2018

@Eh2406 the setup's a little annoying to just explain, so here's a full compiling set of example crates: https://github.com/Nemo157/rust-44663-facade-example

In this example futures-core exports a single trait Future, futures re-exports this trait from futures-core along with a "utility" Foo.

Now mylib wants to use futures, including returning some Futures from its public API. But futures itself intends to have breaking changes to update utilities, users that care about interoperability should only use types from futures-core in their public APIs. Even as new major versions of futures are released they will all link against the same version of futures-core so the underlying traits will be interoperable.

So, to have the compiler enforce this restriction the developers of mylib want to use the public/private dependencies feature, they have both futures and futures-core as dependency but only futures-core marked as public. But for ergonomics it's nicer to just use the re-exports directly from futures, instead of having to remember which parts of the API need to be imported from futures and which from futures-core.

@Aaron1011
Copy link
Member

@Aaron1011 Aaron1011 commented Dec 15, 2018

I'd like to work on this.

@Eh2406
Copy link
Contributor

@Eh2406 Eh2406 commented Dec 16, 2018

Thanks for offering to work on this! @alexcrichton, do you have any mentoring advice?

I can at any time get a branch ready for the resolver part of cargo, it will be good enough to start experimentation, although not good enough to be ready to stabilize.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Dec 17, 2018

Sure! @Aaron1011 so this is split roughly into two features, one being rustc knowing what crates are in the "public interface" and a second being the support in Cargo's resolver to make various decisions differently. Are you interested in one of those in particular?

@Aaron1011
Copy link
Member

@Aaron1011 Aaron1011 commented Dec 17, 2018

@alexcrichton: I've started working on the rustc part in a branch: https://github.com/Aaron1011/rust/commits/feature/pub-priv-dep, and I plan to submit a (WIP) PR soon. I might also be interested in working on the cargo part as well.

@Eh2406
Copy link
Contributor

@Eh2406 Eh2406 commented Dec 17, 2018

@Aaron1011 please cc me when you make that PR. I know nothing about the internals of Rust, so will be of no help, but I do want to follow the discussion.

I will redouble my efforts to get the Cargo's resolver part mergeable. I have been enjoying ( and hating ) it because it is the hardest algorithmic problem I have ever worked on. I would be happy for some help, perhaps a new perspective will get me un-stuck.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Dec 18, 2018

@Aaron1011 ok and it seems like you're off to a great start! I'd be up for reviewing that PR, and if you've got any questions along the way just let me know

@Nemo157
Copy link
Member

@Nemo157 Nemo157 commented Jan 25, 2019

There was another usecase for this mentioned on i.rl.o, if cargo doc had a form of "local development" mode to build the documentation you care about when working on a crate, this would allow that mode to know exactly which dependencies it needs to produce the documentation for (all direct dependencies + their public dependencies).

Similarly, if you're self-hosting documentation for a crate somewhere you may want to be able to generate a documentation bundle including the crate and all its public dependencies so that it is fully self-contained.

bors added a commit that referenced this issue Feb 1, 2019
Implement public/private dependency feature

Implements #44663

The core implementation is done - however, there are a few issues that still need to be resolved:

- [x] The `EXTERNAL_PRIVATE_DEPENDENCY` lint currently does notthing when the `public_private_dependencies` is not enabled. Should mentioning the lint (in an `allow` or `deny` attribute) be an error if the feature is not enabled? (Resolved- the feature was removed)
- [x] Crates with the name `core` and `std` are always marked public, without the need to explcitily specify them on the command line. Is this what we want to do? Do we want to allow`no_std`/`no_core` crates to explicitly control this in some way? (Resolved - private crates are now explicitly specified)
- [x] Should I add additional UI tests? (Resolved - added more tests)
- [x] Does it make sense to be able to allow/deny the `EXTERNAL_PRIVATE_DEPENDENCY` on an individual item? (Resolved - this is implemented)
Aaron1011 added a commit to Aaron1011/crates.io that referenced this issue Mar 21, 2019
This implements the crates.io side of rust-lang/rust#44663

This is fully backwards-compatible - 'public' will default to 'false'
for old versions of Cargo that don't include it in their manifest
@dekellum
Copy link

@dekellum dekellum commented May 10, 2019

I like this feature! The produced warning messages were very clear. In 1 out of 5 crates trialed, I decided based on the results to make 2 dependencies private that were needlessly public. This makes me think there is value and wonder if there is an opportunity to stabilize just the information collection part of this sooner (public dependency attribute, rustc warnings, uploads to the crates.io index) and the dependency resolution changes (warnings or any resolution errors) later? In particular, wouldn't that allow progress on information collection and utility without serializing on rust-lang/cargo#5657? Or perhaps the tracking details just need to be updated here?

Some additional minor feedback below:

% rustc --version
rustc 1.36.0-nightly (a784a8022 2019-05-09)
% cargo --version
cargo 1.36.0-nightly (759b6161a 2019-05-06)
  • If you label something public=true that isn't actually publicly exposed, you currently don't get a warning. Would a warning or lint for that case be an appropriate future addition?

  • With the cargo feature enabled, cargo doc currently fails, due to rustdoc not understanding the --extern-private flags as passed. Workaround is to temporarily remove the feature and all public attributes from deps, or replace --extern-private with --extern and run rustdoc manually.

cargo doc --no-deps --open
 Documenting barc v1.2.0 (/home/david/src/body-image/barc)
error: Unrecognized option: 'extern-private'

error: Could not document `barc`.

Caused by:
  process didn't exit successfully: `rustdoc --edition=2018 --crate-name barc barc/src/lib.rs --color always -o /home/david/src/body-image/target/doc --cfg 'feature="body-image"' --cfg 'feature="brotli"' --cfg 'feature="default"' --cfg 'feature="memmap"' --cfg 'feature="mmap"' --cfg 'feature="olio"' -L dependency=/home/david/src/body-image/target/debug/deps --extern body_image=/home/david/src/body-image/target/debug/deps/libbody_image-f6307513eafde4da.rmeta --extern-private brotli=/home/david/src/body-image/target/debug/deps/libbrotli-914be93b87d00a74.rmeta --extern-private bytes=/home/david/src/body-image/target/debug/deps/libbytes-60c7a2c9551c05c4.rmeta --extern-private flate2=/home/david/src/body-image/target/debug/deps/libflate2-3410ad175205616a.rmeta --extern http=/home/david/src/body-image/target/debug/deps/libhttp-1bcf97f938c6fdd7.rmeta --extern-private httparse=/home/david/src/body-image/target/debug/deps/libhttparse-b04e539fbadba356.rmeta --extern-private memmap=/home/david/src/body-image/target/debug/deps/libmemmap-b29e83fdba55bd43.rmeta --extern-private mime=/home/david/src/body-image/target/debug/deps/libmime-df20e72307836353.rmeta --extern-private olio=/home/david/src/body-image/target/debug/deps/libolio-c7a6754af0be8b56.rmeta --extern-private tao_log=/home/david/src/body-image/target/debug/deps/libtao_log-7da4b359386cbcf7.rmeta --extern-private tempfile=/home/david/src/body-image/target/debug/deps/libtempfile-5c5a144b50e3000b.rmeta -Z unstable-options --cfg barc_std_try_from` (exit code: 1)

I'm happy to elaborate here or elsewhere if its useful.

@Aaron1011
Copy link
Member

@Aaron1011 Aaron1011 commented May 10, 2019

If you label something public=true that isn't actually publicly exposed, you currently don't get a warning. Would a warning or lint for that case be an appropriate future addition?

That sounds like a good idea to me!

Implementing that will be a little tricky. Currently, rustc isn't explicitly aware of the underlying public-dependency feature - it simply treats all dependencies as public by default. We would want to avoid showing 'unecessary public dependency' warnings to users who don't have the public-dependency feature enabled, as these warnings would be both useless and unfixable.

We would probably need to add a -Z public-dependencies flag to rustc to ensure that we only enable the lint when the cargo feature is also enabled.

Is it necessary to make another RFC for something like this?

@dekellum
Copy link

@dekellum dekellum commented May 10, 2019

Or the alternative "--extern(:pub|:private)? name=PATH" syntax proposed by @petrochenkov could also get rustc that detail, if it chose to preserve the distinction for this? It sounds like an implementation detail to me, but what do I know!

@dekellum
Copy link

@dekellum dekellum commented May 15, 2019

I just tried enabling this feature on another lib crate (also for the purpose of collecting information on public deps) and noticed that examples/*.rs are another interesting edge case. If the examples define any pub items (here really just for purpose of example) that reference the lib crate types, these will result in warnings of the pattern:

warning: type `Foo` from private dependency 'libcrate' in public interface

This is because each example is compiled, including as part of cargo test, with --extern-private libcrate=... Note this dependency isn't written in the Cargo.toml, its just implied for examples. Shouldn't examples be compiled with --extern:pub libcrate=... instead (using newer proposed syntax just for clarity)?

Cc: @Eh2406 this last case (and possibly the one above about cargo doc) might be more specific to cargo?

@petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented Oct 15, 2019

The feature's implementation doesn't currently work correctly with transitive dependencies.
If we load a crate named foo as a transitive dependency, we still mark it private (or not) by looking at the --extern foo option, even if refers to an unrelated crate and --extern options in general are entirely irrelevant to transitive dependencies.

The dependency private-ness should be written into metadata and read from there for transitively loaded crates.

This brings a question - what if the same crate is private direct dependency and a public transitive dependency (or vice versa)? Should it be treated as private or public?

(I'm currently refactoring code in that area and may fix this issue as well.)

@petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented Oct 15, 2019

@Aaron1011
By the way, do you still have plans to implement the --extern:modifier syntax from #44663 (comment)?

@ehuss
Copy link
Contributor

@ehuss ehuss commented Oct 15, 2019

@petrochenkov I like the idea of the modifier syntax, but I'm curious how that would work with getopts. If the intent would be to support multiple modifiers, would every permutation need to be registered as separate flags?

@petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented Oct 15, 2019

how that would work with getopts.

No idea :)
I didn't look into implementation details, only suggested a syntax that's more generic than --extern-private.

@ehuss
Copy link
Contributor

@ehuss ehuss commented Dec 6, 2019

I have posted a proposal for a new --extern flag syntax at #67074.

@ehuss
Copy link
Contributor

@ehuss ehuss commented Dec 7, 2019

I'm a bit confused about how re-exports are being treated. From the RFC: "Q: Can I export a type from a private dependency as my own?" says NO. However, it looks like various ways of re-exporting do not generate a warning:

  • pub extern crate private_crate; or pub use private_crate;
  • pub use private_crate::some_item;

Is that intentional?

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Jul 17, 2021

@Eh2406 in rust-lang/rfcs#3146 (comment) wrote

The problem with "Public and private dependencies" is not the theory. It is fairly clear what we want it to mean. It is implementation that is the problem. In a SAT/SMT reduction of Dependency resolution it adds O( (numbers of packages considered) ^ 2) terms to the problem. More importantly in the Resolver as implemented it takes common use cases from sub second time to multiple hours time. There are a lot of changes to the Resolver that we can not stabilize because of limitations to performance or error messages even though we have a clear understanding of exactly what behavior we want and why.

Is more written about this? It definitely does make it harder, but I am surprised it is so bad in practice when Cabal etc. do this today. Were you ret-conning existing deps as public deps? (I thought the plan was to keep them private which I have my doubts about but which I thought would not suddenly increase the costs.)

@Eh2406
Copy link
Contributor

@Eh2406 Eh2406 commented Jul 18, 2021

I wrote that from memory, and wanted to go back and reread the records (mostly rust-lang/cargo#6129) before expanding on it. It seems like my memory had lost some of the subtlety of the situation. Sorry.

There are some open question about behavior, best somerized at rust-lang/cargo#6129 (comment). However, except for important surface level sintacs discussions, the available design space is pretty small. For various reasons there needs to be a "existing behavior" option in addition to "public" or "private" dependencies.

As to the runtime overhead for existing dependencies it is not large. As everything is marked "existing behavior", so it can create no conflicts. Even in the current implementation where everything is marked as private, which means that this can only add a conflict for the parent, conflicts tend to be small and simple. However the Resolver is a tricky and organically grown peace of code. I am not willing to stabilize new behavior for it that is not end-to-end property tested. If you turn on public & private dependencies in our test case generator here, it will find a case where things take over 60 seconds to resolve. If you comment out that line or run the tests in release, it will find a case where our organically grown form of learned clauses has over simplified the problem, so the Resolver decided there is no valid solution, when there is one and the "Reduction to SAT" test can find it. I burnt myself out trying to find the sweet spot, a learned clause representation sparse enough for efficient back jumping with enough information to not miss cases.

Since then, I have been spending my time on the PubGrub project in the hope that by copying from the state of the art and rewriting from first principles, we will have a more solid foundation for adding these kinds of features.

The RFC pre dates the "Prior art" section. Do you have links to info about the corresponding features in "Cabal etc."?

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Aug 3, 2021

@Eh2406 Sorry to leave your nice response hanging.

The RFC pre dates the "Prior art" section. Do you have links to info about the corresponding features in "Cabal etc."?

Off hand, just https://wiki.haskell.org/wikiupload/b/b4/HIW2011-Talk-Loeh.pdf linked in the other thread, and I found https://opam.ocaml.org/doc/External_solvers.html for OPAM which links some older research. But I would be happy to ask around to get more up to date info for Cabal.

Since then, I have been spending my time on the PubGrub project in the hope that by copying from the state of the art and rewriting from first principles, we will have a more solid foundation for adding these kinds of features.v

I read up a bit on that. Interesting and looks quite nice! It does seem the PubGrub algorithm out of the box is just for the all-public-deps case right? So if we switched to that it would be the solid foundation and the new features together?

@Eh2406
Copy link
Contributor

@Eh2406 Eh2406 commented Aug 4, 2021

PubGrub simplifies dependency resolution to a platonic ideal of it self. There is a lot of work to expand it to handle compatibility with all the gnarly details of Cargos resolver. A full discussion of each problem and are current thoughts for how to work around it is somewhat out of scope here, luckily @mpizenberg has written up a guide here with a section on "Advanced usage and limitations". We have a plan for how to reduce Cargo's "one per name and semver compatibility range" to PubGrub's "one per name". Figuring out how to extend that to also have "scoped goals" or "public dependencies", has not yet been figured out.

Despite having to do a lot of work to get back to where we are, I am hopeful that building off this base will work out better because PubGrub simplifications run deep. It has combined many different concepts (that Cargo's resolver has as different types) into one comparatively simple "incompatibility" type. All the different aspects of the problem (Error reporting, Conflict resolution, Unit propagation, ... ) share that vocabulary type. In Cargo's resolver adding a new "learned clause" kind involves interacting the new enum variant with all the moving parts, and thanks to the end-to-end property tests finding all the corner cases where they interact badly. I am hopeful that in PubGrub it is "express your property's as incompatibilities" (ether by reduction to the existing simplified problem, or by adding a new API) and then all the existing pieces will work with it.

Maybe someone will find a way to add it before the rebuilding compleats, but PubGrub is where I will be spending my time for now.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Aug 8, 2021

Maybe someone will find a way to add it before the rebuilding compleats, but PubGrub is where I will be spending my time for now.

Yes I definitely agree with this approach, to be clear.

We have a plan for how to reduce Cargo's "one per name and semver compatibility range" to PubGrub's "one per name". Figuring out how to extend that to also have "scoped goals" or "public dependencies", has not yet been figured out.

Heh that's the tricky bit. I would say "one per name" means all public dependencies. Cargo's "one per name and semver compatibility range" to me feel more like an optimization that all-private/unrestricted deps don't cause too much duplicative bloat -- I can't see how having multiple versions that that are semver compatible would actually break things, just cause a nuscience.

"scoped goals" sounds like private dependencies to me. Or rather, it's the combination of public and private deps that induces local non-global coherence issues. (All private means solving is trivial, all public means PubGrub as written works.)

I do see how one can turn foo = ">5" into foo-6 = * or foo-7 = * or ..., but I worry it's a bit of a dead end. By all means, do something like that for now if you gets Cargo on PubGrub sooner, but I think changes will need to be made to PubGrub to actually track how packages are instantiated. Trying to defunctionalize something like foo ^>= 7 & bar(foo) ^>= 9 gets too messy.

@mpizenberg
Copy link

@mpizenberg mpizenberg commented Aug 8, 2021

Hi @Ericson2314 I'm just adding a few notes regarding PubGrub. We have to think more about the public/private feature, but I think it will be doable with a "proxy and bucket" scheme similar to the one presented in the guide. Anyway, we'll have a look at it and report how that goes.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Aug 18, 2021

@mpizenberg Thanks for that info! One thing I am slightly unclear on is how the "coherence" of public deps is maintained. I think you might need equality constraints, like a->c = a->b->c for the case where a has a public dep on b and c, and b also has a public dep on c.

@mpizenberg
Copy link

@mpizenberg mpizenberg commented Aug 19, 2021

@Ericson2314 for more details, I've written a new section in the guide specifically about implementing public/private dependencies. It is currently under review with @Eh2406 so it's not published yet. But if you want to participate in the review of this section it lives here: pubgrub-rs/guide#4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet