New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build-dependencies and dependencies should not have features unified #4866

Open
whitequark opened this Issue Dec 27, 2017 · 25 comments

Comments

Projects
None yet
10 participants
@whitequark
Copy link
Member

whitequark commented Dec 27, 2017

Consider this real-world example from crc-rs:

[dependencies]
build_const = { version = "0.2", default-features = false }

[build-dependencies]
build_const = "0.2"

The build dependency, of course, needs (and has) the std feature. The runtime dependency does not. Moreover, on my platform, there is no std. However, cargo treats them as the same package, and as a result it is impossible to actually build crc-rs at all.

@vitiral

This comment has been minimized.

Copy link

vitiral commented Dec 30, 2017

wow, I certainly would not have anticipated this (and appear to have not tested it...). Yay me!

@gnzlbg

This comment has been minimized.

Copy link

gnzlbg commented Feb 4, 2018

Not only dev-dependencies, features of platform-specific dependencies are also unified:

With this Cargo.toml:

[package]
name = "cargo_fubar"
version = "0.1.0"
authors = ["fubar <fubar@fubar.com>"]

[target.'cfg(target_os = "macos")'.dependencies]
mach = "0.1.*"

[dependencies.libc]
version = "0.2"
default-features = false

cargo build --no-default-features --verbose --target x86_64-unknown-linux-gnu outputs:

 Compiling libc v0.2.36
     Running `rustc --crate-name libc /foo/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.36/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=af3eb212fae2904c -C extra-filename=-af3eb212fae2904c --out-dir /foo/projects/sideprojects/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -L dependency=/foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/foo/cargo_fubar/target/debug/deps --cap-lints allow`

Note how I have a platform-specific dependency for MacOSX which should absolutely have nothing to do with linux builds, yet --cfg 'feature="default"' --cfg 'feature="use_std"' is enabled for libc even though [dependencies.libc] and the cargo build --no-default-features explicitly disable all features.

Removing the platform specific dependency and recompiling for linux produces:

Compiling libc v0.2.36
     Running `rustc --crate-name libc /foo/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.36/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=ebd4a22423d44421 -C extra-filename=-ebd4a22423d44421 --out-dir /foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps --target x86_64-unknown-linux-gnu -L dependency=/foo/cargo_fubar/target/x86_64-unknown-linux-gnu/debug/deps -L dependency=/foo/cargo_fubar/target/debug/deps --cap-lints allow`

Any workarounds?

@gnzlbg

This comment has been minimized.

Copy link

gnzlbg commented Feb 4, 2018

@alexcrichton I disagree about this being a feature request, I think this is a pretty severe bug in cargo and should be tagged as such.

@gnzlbg

This comment has been minimized.

Copy link

gnzlbg commented Feb 4, 2018

So I've added a test for this in PR #5007. If nobody beats me to it (and I hope somebody will) I'll either submit a new PR with a fix or add it to that PR (if it doesn't get merged).

@gnzlbg

This comment has been minimized.

Copy link

gnzlbg commented Feb 5, 2018

So fixing this is very hard, it looks like cargo was designed under the assumption that the features of all dependencies, dev-dependencies, target.dependencies,.... ought to be unified to be able to generate a single dependency graph for build, test, --target, etc.

What I've ended up doing as a workaround is having multiple Cargo.toml files for a single project... I have a Cargo_test.toml, Cargo_build_target_xxx.toml, etc. This sucks, but it work. However, I can't release to crates.io under this setup, because I don't have a single Cargo.toml for all my targets, and adding multiple targets messes the features...

@gnzlbg

This comment has been minimized.

Copy link

gnzlbg commented Feb 5, 2018

Someone should raise this with the tools team. I don't know how anybody manages to use cargo for targeting any #[no_std] environment.

If your crate happens to use std in any way for, for example, testing or benchmarking, chances are its always being built with std linked in, and forcing other crates to be linked with std as well :/

@vitiral

This comment has been minimized.

Copy link

vitiral commented Feb 5, 2018

@gnzlbg that's an interesting workaround, however it wouldn't fix the issue if your build.rs needed it.

For testing/benchmarking only could you have Cargo.toml be your "released application" and CargoTest.toml be your "test" one?

@SimonSapin

This comment has been minimized.

Copy link
Contributor

SimonSapin commented Mar 9, 2018

I think this is related to #4463.

@Ericson2314

This comment has been minimized.

Copy link
Contributor

Ericson2314 commented Mar 18, 2018

Yeah this is crucial blocker for cross compilation. The solution is a huge refactor where each build.rs generates a new solution space (or at the very least one for each build.rs, one for each build.rs needed by a level-1 build.rs, and so on).

If it helps Haskell's Cabal has some great prior art here.

  1. The library, binaries, and build.rs of a package are viewed as separate "components", totally separate nodes in the dependency graph with their own distinct dependencies, but not totally separate nodes in the solution space since they are required to come from the same package.
  2. qualified goals are unresolved deps along with just enough provenance information to indicate which simultaneous solution we are solving for. For example, if every build.rs gets independent versions, we could use Vec<Package> for the chain of build.rs, or if we separate build.rs into stages then we just need a counter (corresponding to the length of the chain).

Lastly, this all becomes far more important with public dependencies, when it's required that similar public dependency edges point to the same node. Separately solution spaces are the one way to break those constraints.

Also, as #4866 (comment) sadly demonstrates, the original idea that is a single solution in a Cargo.lock can work for all platforms is probably unworkable. Whereas we can in always solve build.rs separately for consistency, platform-specific crate- and feature- dependencies can "wildly" change the solution space. CC @wycats.

[Ironically, I do think we can eventually simultaneously and cheaply type-check each crate against all possible dependencies (!!!), so perhaps this isn't so bad. Even if the lockfile contains only a few canonical plans for different platforms, we can ensure that all others will at least typecheck!]

@jethrogb jethrogb referenced this issue Mar 18, 2018

Open

Cargo improvements and bugs #5

1 of 4 tasks complete
@gnzlbg

This comment has been minimized.

Copy link

gnzlbg commented Mar 18, 2018

@Ericson2314 To really solve this I think that cargo would need to support a dependency graph per profile.

Huge refactor is probably an understatement. We would not only need a backwards compatible mode to support old Cargo.lock files, but we will also probably want to unify the dependency graphs or sub trees of them when possible to avoid downloading and compiling a potentially different version of a dependency per profile.

Also, this will also mean that cargo bench might use a very different dependency graph from cargo build --release, and probably many other things that will need to be worked out.

Also, the solution might be RFC worthy :/

@Ericson2314

This comment has been minimized.

Copy link
Contributor

Ericson2314 commented Mar 19, 2018

Uhhhh I'm not really sure what profiles are (though I think that goes for most of us), but I wouldn't really expect them to need their own graphs in general? Debug vs release should explicitly be the same thing. system test are really just binaries with a special purpose. The best example would be unit tests since they are turning the primary component (whatever it may be) into a binary.

In otherwords, I think components + optimizations is a much better model than this hopelessly overloaded concept of "profiles".

@Ericson2314

This comment has been minimized.

Copy link
Contributor

Ericson2314 commented Mar 19, 2018

@gnzlbg I hope old lock files can at least be read only.

@gnzlbg

This comment has been minimized.

Copy link

gnzlbg commented Mar 19, 2018

but I wouldn't really expect them to need their own graphs in general?

I meant, if we are going to move cargo from one dependency graph to two dependency graphs, one for dev builds, and one for non-dev builds, we might as well do so in a way that supports n dependency graphs (worst case one per profile).

Some people have requested "dev-only" features, others have requested ways to turn on/off "unstable" feature, the way to customize things in config.toml increases, others have requested custom profiles, potentially with profile-dependent dependencies, etc.

The problem we have right now is that cargo is designed towards a single dependency graph. Even if we only end up with two after fixing this, the solution should be able to handle n graphs.

The best example would be unit tests since they are turning the primary component (whatever it may be) into a binary.

So the reason cargo bench and cargo test --release might have different dependencies than cargo build --release is that bench and test use the dev-profile. That is, for benchmarking and testing you might have extra crates (only used by the tests or benchmarks), and just by adding these extra crates to your graph you are potentially changing the whole graph (e.g. version numbers).

@gnzlbg I hope old lock files can at least be read only.

I think this is a must - it's part of backwards compatibility. It is just another constraint that one needs to keep in mind when trying to fix this.

@Ericson2314

This comment has been minimized.

Copy link
Contributor

Ericson2314 commented Mar 19, 2018

@gnzlbg

we might as well do so in a way that supports n dependency graphs

OK yeah agreed on not hard-coding a fixed number for sure.

Some people have requested "dev-only" features...

I've see a few those requests; IMO profiles are such a broken concept people should resubmit their requests in more general terms so we can start over.

So the reason cargo bench and cargo test --release might have different dependencies...

Yeah so when we're building different binaries (bench binaries, test binaries) it makes total sense to me that we have a different dependency graph. Less clear to me is that should be built with a changed library (assuming the package isn't just binaries): arguably, cfg(test) in the main library should just be for building unit tests (by which I've meant the internal tests with the test runner), external tests should get the same library (and the optimized cargo build --release version for external benchmarks). Basically this is what Cabal does and I find it makes far more sense. (OTOH Cabal has no good support for the internal tests/benches, and that's a huge mess in Haskell land.)

I think this is a must - it's part of backwards compatibility. It is just another constraint that one needs to keep in mind when trying to fix this.

Ah sorry my "at least" would probably have been better as an "at most". I would definitely not want to support writing old Cargo.toml; one would have to convert it into the new format first.

@stale

This comment has been minimized.

Copy link

stale bot commented Sep 15, 2018

As there hasn't been any activity here in over 180 days I've marked this as stale and if no further activity happens for 7 days I'll will close it.

I'm a bot so this may be in error! If this issue should remain open, could someone (the author, a team member, or any interested party) please comment to that effect?

The team would be especially grateful if such a comment included details such as:

  • Is this still relevant?
  • If so, what is blocking it?
  • Is it known what could be done to help move this forward?

Thank you for contributing!

If you're reading this comment from the distant future, fear not if this was closed automatically. If you believe it's still an issue please leave a comment and a team member can reopen this issue. Opening a new issue is also acceptable!

@stale stale bot added the stale label Sep 15, 2018

@Ericson2314

This comment has been minimized.

Copy link
Contributor

Ericson2314 commented Sep 15, 2018

This is still plenty relevant, but hasn't been prioritized in any fashion.

@stale stale bot removed the stale label Sep 15, 2018

dwijnand added a commit to dwijnand/cargo that referenced this issue Sep 15, 2018

Add newline spacing to stale bot's message
So our first incantation produced:

rust-lang#4866 (comment)

Perhaps there's a better indicator to use, but for now let's just fix
this with more newlines.

r? @alexcrichton

bors added a commit that referenced this issue Sep 15, 2018

Auto merge of #6030 - dwijnand:tweak-stale-bot-message, r=alexcrichton
Add newline spacing to stale bot's message

So our first incantation produced:

#4866 (comment)

Perhaps there's a better indicator to use, but for now let's just fix
this with more newlines.

r? @alexcrichton
@yaram

This comment has been minimized.

Copy link

yaram commented Nov 23, 2018

I'm facing this issue at the moment. Would be nice if it got some attention.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Jan 27, 2019

I've been looking into this problem. I believe the feature resolver is here?

https://github.com/rust-lang/cargo/blob/bd536c6/src/cargo/core/resolver/context.rs#L158

    /// Return all dependencies and the features we want from them.
    fn resolve_features<'b>(
        &mut self,
        parent: Option<&Summary>,
        s: &'b Summary,
        method: &'b Method<'_>,
    ) -> ActivateResult<Vec<(Dependency, Vec<InternedString>)>> {
        let dev_deps = match *method {
            Method::Everything => true,
            Method::Required { dev_deps, .. } => dev_deps,
        };

        // First, filter by dev-dependencies
        let deps = s.dependencies();
        let deps = deps.iter().filter(|d| d.is_transitive() || dev_deps);

        let reqs = build_requirements(s, method)?;
        let mut ret = Vec::new();
        let mut used_features = HashSet::new();
        let default_dep = (false, Vec::new());

        // Next, collect all actually enabled dependencies and their features.
        for dep in deps {
            [...]
        }
        [...]
    }

If this is the correct resolver, it does appear to unilaterally incorporate the build and dev dependencies into the feature calculation. However, it appears there's one conditional gate on required dev dependencies vs Everything, which I assume also considers optional dev dependencies?

@Eh2406

This comment has been minimized.

Copy link
Contributor

Eh2406 commented Jan 27, 2019

I think that is correct, specifically that code is looking at all features that are activated for that Summary then filtering out all dev dependencies of transitive crates.

I think this is a request that this dependency get built more then once. I think there are deeper structural problems that will have to be worked out to make this happen. There are lots of things that assume that the "list of things to build" can be indexed by package_id. Most directly the hashmap from package_id to the list of features to build it with. will need a way to have 2 entries, one for build_const with default-features and one without. A similar fix will need to be make to every other HashMap<PackageId in that Resolve struct. That will require similar changes in the RegistryQueryer, Context, and the Conflict system and Cache. That is just the changes in core/resolver. I believe that that assumption is baked into large amounts of the rest of the code as well, but don't know it well enough to find links.

@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Jan 27, 2019

It's not just the code. There are high-level design decisions to be made. I'm not yet convinced just separating dev-dependencies is the right direction. Some projects specifically want features to be unified (rustc-workspace-hack). I'm also concerned that will break existing projects. There are a variety of issues with features and dependency unification, and I think it would be best to have a picture of what it should ultimately look like before attacking it piecemeal.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Jan 27, 2019

The biggest issue at hand is this is an extremely confusing failure mode for no_std developers, and one that I see popping up again and again (here's the latest in the past 24 hours):

dalek-cryptography/curve25519-dalek#187 (comment)

Where in other cases spurious feature activation wouldn't matter, no_std relies on every single dependency not linking std. This is already hard to debug as out-of-the-box rustc provides little indication which specific crates are linking std (fortunately 3rd party tools like cargo nono and cargo tree can help), so having the std feature activated in a project's dependencies via spooky-action-at-a-distance from build-dependencies and dev-dependencies makes for extremely confusing errors.

Concretely the following would make sense to me:

  1. build-dependencies should not activate cargo features in the target whatsoever
  2. dev-dependencies should not activate cargo features in release builds

The first issue is exceedingly common with tools like bindgen which are large command line applications with numerous dependencies. Something I was spitballing about was a potential workaround was whether or not these sorts of tools could get cargo installed as part of the build process so they could sidestep the project's dependency resolution.

The second issue pops up frequently when std is used in tests but not release builds, especially transitively via a dependency used only for tests.

Either way this is a nightmare for both no_std users and authors of crates which are advertised as being no_std compatible, but suffer these sorts of breakages which dependencies which at first glance should not activate std in the target end up doing so accidentally.

@tarcieri

This comment has been minimized.

Copy link

tarcieri commented Jan 27, 2019

I think this is a request that this dependency get built more then once. I think there are deeper structural problems that will have to be worked out to make this happen. There are lots of things that assume that the "list of things to build" can be indexed by package_id.

I got lost trying to trace how this all works, but something I've been wondering about is how cargo workspaces provide a common dependency resolution in Cargo.lock, but are able to activate specific dependencies and features per package within the workspace.

Could build-dependencies and proc macros for a particular package use a sort of "virtual workspace" which is independent of the target's dependency and feature resolution?

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jan 28, 2019

I could have sworn I'd already written up instructions for how this might be tackled at a high level, but I can't seem to find them now. Regardless both @Eh2406 and @ehuss are right. This is a pretty deep change to how Cargo works and doesn't have a targeted fix (trust me, if we could think of an easy fix after this many years we would have implemented it). This is also a pretty big change to Cargo and needs to be considered carefully and measured for its impact on the ecosystem.

I personally believe, however, that we need to do all we can to close out this an related issues. I'm a fan of a model where dependency edges activate features but only within the context of a particular target. After resolution, when we're actually building crates, we'd enable features only for a particular target and that would produce the desired behavior here.

Now that I'm thinking about this @ehuss didn't you have an excellent writeup about this somewhere as well? We may want to invest a bit of time deduplicating all these issues...

@ehuss

This comment has been minimized.

Copy link
Contributor

ehuss commented Jan 28, 2019

@alexcrichton Perhaps you're thinking of #5730 (comment) and rust-lang/crates-io-cargo-teams#13 (comment)?

I have categorized all the issues in my notes. There are some duplicates, but I think they all contain some useful information and subtle differences.

@alexcrichton

This comment has been minimized.

Copy link
Member

alexcrichton commented Jan 28, 2019

Ah I believe those are indeed what I'm thinking of, yes! And seems fine to me to leave open if they're different!

@dwijnand dwijnand changed the title dev-dependencies and dependencies should not have features unified build-dependencies and dependencies should not have features unified Feb 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment