Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build-dependencies and dependencies should not have features unified #4866

Closed
whitequark opened this issue Dec 27, 2017 · 40 comments
Closed

build-dependencies and dependencies should not have features unified #4866

whitequark opened this issue Dec 27, 2017 · 40 comments

Comments

@whitequark
Copy link
Member

@whitequark whitequark commented Dec 27, 2017

Consider this real-world example from crc-rs:

[dependencies]
build_const = { version = "0.2", default-features = false }

[build-dependencies]
build_const = "0.2"

The build dependency, of course, needs (and has) the std feature. The runtime dependency does not. Moreover, on my platform, there is no std. However, cargo treats them as the same package, and as a result it is impossible to actually build crc-rs at all.

@vitiral
Copy link

@vitiral vitiral commented Dec 30, 2017

wow, I certainly would not have anticipated this (and appear to have not tested it...). Yay me!

@gnzlbg
Copy link

@gnzlbg gnzlbg commented Feb 4, 2018

Not only dev-dependencies, features of platform-specific dependencies are also unified:

With this Cargo.toml:

[package]
name = "cargo_fubar"
version = "0.1.0"
authors = ["fubar <fubar@fubar.com>"]

[target.'cfg(target_os = "macos")'.dependencies]
mach = "0.1.*"

[dependencies.libc]
version = "0.2"
default-features = false

cargo build --no-default-features --verbose --target x86_64-unknown-linux-gnu outputs:

 Compiling libc v0.2.36
     Running `rustc --crate-name libc /foo/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.36/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=af3eb212fae2904c -C extra-filename=-af3eb212fae2904c --out-dir /foo/projects/sideprojects/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -L dependency=/foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/foo/cargo_fubar/target/debug/deps --cap-lints allow`

Note how I have a platform-specific dependency for MacOSX which should absolutely have nothing to do with linux builds, yet --cfg 'feature="default"' --cfg 'feature="use_std"' is enabled for libc even though [dependencies.libc] and the cargo build --no-default-features explicitly disable all features.

Removing the platform specific dependency and recompiling for linux produces:

Compiling libc v0.2.36
     Running `rustc --crate-name libc /foo/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.36/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=ebd4a22423d44421 -C extra-filename=-ebd4a22423d44421 --out-dir /foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -L dependency=/foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/foo/cargo_fubar/target/debug/deps --cap-lints allow`

Any workarounds?

@gnzlbg
Copy link

@gnzlbg gnzlbg commented Feb 4, 2018

@alexcrichton I disagree about this being a feature request, I think this is a pretty severe bug in cargo and should be tagged as such.

@gnzlbg
Copy link

@gnzlbg gnzlbg commented Feb 4, 2018

So I've added a test for this in PR #5007. If nobody beats me to it (and I hope somebody will) I'll either submit a new PR with a fix or add it to that PR (if it doesn't get merged).

@gnzlbg
Copy link

@gnzlbg gnzlbg commented Feb 5, 2018

So fixing this is very hard, it looks like cargo was designed under the assumption that the features of all dependencies, dev-dependencies, target.dependencies,.... ought to be unified to be able to generate a single dependency graph for build, test, --target, etc.

What I've ended up doing as a workaround is having multiple Cargo.toml files for a single project... I have a Cargo_test.toml, Cargo_build_target_xxx.toml, etc. This sucks, but it work. However, I can't release to crates.io under this setup, because I don't have a single Cargo.toml for all my targets, and adding multiple targets messes the features...

@gnzlbg
Copy link

@gnzlbg gnzlbg commented Feb 5, 2018

Someone should raise this with the tools team. I don't know how anybody manages to use cargo for targeting any #[no_std] environment.

If your crate happens to use std in any way for, for example, testing or benchmarking, chances are its always being built with std linked in, and forcing other crates to be linked with std as well :/

@vitiral
Copy link

@vitiral vitiral commented Feb 5, 2018

@gnzlbg that's an interesting workaround, however it wouldn't fix the issue if your build.rs needed it.

For testing/benchmarking only could you have Cargo.toml be your "released application" and CargoTest.toml be your "test" one?

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Mar 9, 2018

I think this is related to #4463.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Mar 18, 2018

Yeah this is crucial blocker for cross compilation. The solution is a huge refactor where each build.rs generates a new solution space (or at the very least one for each build.rs, one for each build.rs needed by a level-1 build.rs, and so on).

If it helps Haskell's Cabal has some great prior art here.

  1. The library, binaries, and build.rs of a package are viewed as separate "components", totally separate nodes in the dependency graph with their own distinct dependencies, but not totally separate nodes in the solution space since they are required to come from the same package.
  2. qualified goals are unresolved deps along with just enough provenance information to indicate which simultaneous solution we are solving for. For example, if every build.rs gets independent versions, we could use Vec<Package> for the chain of build.rs, or if we separate build.rs into stages then we just need a counter (corresponding to the length of the chain).

Lastly, this all becomes far more important with public dependencies, when it's required that similar public dependency edges point to the same node. Separately solution spaces are the one way to break those constraints.

Also, as #4866 (comment) sadly demonstrates, the original idea that is a single solution in a Cargo.lock can work for all platforms is probably unworkable. Whereas we can in always solve build.rs separately for consistency, platform-specific crate- and feature- dependencies can "wildly" change the solution space. CC @wycats.

[Ironically, I do think we can eventually simultaneously and cheaply type-check each crate against all possible dependencies (!!!), so perhaps this isn't so bad. Even if the lockfile contains only a few canonical plans for different platforms, we can ensure that all others will at least typecheck!]

@gnzlbg
Copy link

@gnzlbg gnzlbg commented Mar 18, 2018

@Ericson2314 To really solve this I think that cargo would need to support a dependency graph per profile.

Huge refactor is probably an understatement. We would not only need a backwards compatible mode to support old Cargo.lock files, but we will also probably want to unify the dependency graphs or sub trees of them when possible to avoid downloading and compiling a potentially different version of a dependency per profile.

Also, this will also mean that cargo bench might use a very different dependency graph from cargo build --release, and probably many other things that will need to be worked out.

Also, the solution might be RFC worthy :/

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Mar 19, 2018

Uhhhh I'm not really sure what profiles are (though I think that goes for most of us), but I wouldn't really expect them to need their own graphs in general? Debug vs release should explicitly be the same thing. system test are really just binaries with a special purpose. The best example would be unit tests since they are turning the primary component (whatever it may be) into a binary.

In otherwords, I think components + optimizations is a much better model than this hopelessly overloaded concept of "profiles".

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Mar 19, 2018

@gnzlbg I hope old lock files can at least be read only.

@gnzlbg
Copy link

@gnzlbg gnzlbg commented Mar 19, 2018

but I wouldn't really expect them to need their own graphs in general?

I meant, if we are going to move cargo from one dependency graph to two dependency graphs, one for dev builds, and one for non-dev builds, we might as well do so in a way that supports n dependency graphs (worst case one per profile).

Some people have requested "dev-only" features, others have requested ways to turn on/off "unstable" feature, the way to customize things in config.toml increases, others have requested custom profiles, potentially with profile-dependent dependencies, etc.

The problem we have right now is that cargo is designed towards a single dependency graph. Even if we only end up with two after fixing this, the solution should be able to handle n graphs.

The best example would be unit tests since they are turning the primary component (whatever it may be) into a binary.

So the reason cargo bench and cargo test --release might have different dependencies than cargo build --release is that bench and test use the dev-profile. That is, for benchmarking and testing you might have extra crates (only used by the tests or benchmarks), and just by adding these extra crates to your graph you are potentially changing the whole graph (e.g. version numbers).

@gnzlbg I hope old lock files can at least be read only.

I think this is a must - it's part of backwards compatibility. It is just another constraint that one needs to keep in mind when trying to fix this.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Mar 19, 2018

@gnzlbg

we might as well do so in a way that supports n dependency graphs

OK yeah agreed on not hard-coding a fixed number for sure.

Some people have requested "dev-only" features...

I've see a few those requests; IMO profiles are such a broken concept people should resubmit their requests in more general terms so we can start over.

So the reason cargo bench and cargo test --release might have different dependencies...

Yeah so when we're building different binaries (bench binaries, test binaries) it makes total sense to me that we have a different dependency graph. Less clear to me is that should be built with a changed library (assuming the package isn't just binaries): arguably, cfg(test) in the main library should just be for building unit tests (by which I've meant the internal tests with the test runner), external tests should get the same library (and the optimized cargo build --release version for external benchmarks). Basically this is what Cabal does and I find it makes far more sense. (OTOH Cabal has no good support for the internal tests/benches, and that's a huge mess in Haskell land.)

I think this is a must - it's part of backwards compatibility. It is just another constraint that one needs to keep in mind when trying to fix this.

Ah sorry my "at least" would probably have been better as an "at most". I would definitely not want to support writing old Cargo.toml; one would have to convert it into the new format first.

@stale
Copy link

@stale stale bot commented Sep 15, 2018

As there hasn't been any activity here in over 180 days I've marked this as stale and if no further activity happens for 7 days I'll will close it.

I'm a bot so this may be in error! If this issue should remain open, could someone (the author, a team member, or any interested party) please comment to that effect?

The team would be especially grateful if such a comment included details such as:

  • Is this still relevant?
  • If so, what is blocking it?
  • Is it known what could be done to help move this forward?

Thank you for contributing!

If you're reading this comment from the distant future, fear not if this was closed automatically. If you believe it's still an issue please leave a comment and a team member can reopen this issue. Opening a new issue is also acceptable!

@ehuss
Copy link
Contributor

@ehuss ehuss commented Feb 24, 2020

Non-unification of build-dependency features has been implemented and is available as a nightly-only feature on the latest nightly 2020-02-23. See the tracking issue at #7915.

If people following this issue could try it out, and leave your feedback on the tracking issue (#7915), I would appreciate it. Particularly we'd like to know if it helps your project, does it cause any breakage, and does it significantly increase initial compile time.

@ehuss ehuss closed this Feb 24, 2020
@valarauca valarauca mentioned this issue Feb 24, 2020
1 of 4 tasks complete
@valarauca
Copy link

@valarauca valarauca commented Feb 24, 2020

Could this issue be re-opened? #7820 doesn't effect proc macro's which this issue reports.

@ehuss
Copy link
Contributor

@ehuss ehuss commented Feb 24, 2020

@valarauca how so? The title says "build-dependencies" and the example is [build-dependencies]. I don't see any comments mentioning macros here.

FWIW, I don't think you need to be worried about proc-macros being forgotten. They will be tracked in #7915, and new issues will be added if needed down the road.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Feb 25, 2020

Well, it's the same principle that applies to proc macros too:

If an observer and observe choice a and b, and do more thing when a = b, either solving must constrain a = b (coherence), or the observer must be constrained to not be able to compare or observe the equivalence of a and b (skolemization)

Regardless of what issues are renamed/reopened/whatever, it would be nice to have the principle articulated. Besides proc macros, it's the same principle that necessitates having public vs private dependencies, and how Cargo interprets them, too.

@gopakumarce
Copy link

@gopakumarce gopakumarce commented Mar 11, 2021

Is this issue fixed for regular dependencies ? I see that the below config makes non-android targets to also pass in the vendored flag :(

[target.'cfg(target_os = "android")'.dependencies]
native-tls = {version = "0.2", features = ["vendored"]}

[target.'cfg(not(target_os = "android"))'.dependencies]
native-tls = "0.2"
@mbrubeck
Copy link
Contributor

@mbrubeck mbrubeck commented Mar 11, 2021

The fix for this issue is shipping in Rust 1.51 (currently on the beta channel, coming to the stable channel on March 25th), and requires opting in to the new feature resolver.

@gopakumarce
Copy link

@gopakumarce gopakumarce commented Mar 11, 2021

thx for the response, so can I get that fix if I use the nightly version ?

@mbrubeck
Copy link
Contributor

@mbrubeck mbrubeck commented Mar 11, 2021

thx for the response, so can I get that fix if I use the nightly version ?

Yes. To get the fix, you must use the nightly or beta toolchain, and add resolver = "2" to the [package] section of your Cargo.toml.

@golddranks
Copy link
Contributor

@golddranks golddranks commented Mar 11, 2021

@gopakumarce The fixes that are talked about are already in the beta version, so no need to use nightly at the moment.

@gopakumarce
Copy link

@gopakumarce gopakumarce commented Mar 11, 2021

thanks a lot !! I will try it

@gopakumarce
Copy link

@gopakumarce gopakumarce commented Mar 12, 2021

warning: resolver for the non root package will be ignored, specify resolver at the workspace root:

But my root Cargo.toml specifies a [workspace] collection of other rust packages and i cant add a package section there! So what do I do :)

@ehuss
Copy link
Contributor

@ehuss ehuss commented Mar 12, 2021

@gopakumarce You can specify the resolver version in the [workspace] table. More documentation is here: https://doc.rust-lang.org/beta/cargo/reference/resolver.html#resolver-versions

@gopakumarce
Copy link

@gopakumarce gopakumarce commented Mar 12, 2021

awesome, that works .. excited to have this feature .. allows me to add openssl on android and just use the native-tls on ios/windows!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet