Feature selection in workspace depends on the set of packages compiled

[matklad]

Reproduction:

1. Check out this commit: matklad/fall@3022be4

2. Build some test with cargo test -p fall_test -p fall_test -p lang_rust -p lang_rust -p lang_json --verbose --no-run

3. Build other tests with cargo test --all --verbose --no-run

4. Run cargo test -p fall_test -p fall_test -p lang_rust -p lang_rust -p lang_json --verbose --no-run again and observe that memchr and some other dependencies are recompiled.

5. Run cargo test --all --verbose --no-run and observe memchr recompiled again.

The verbose flag gives the following commands for memchr:

Running `rustc --crate-name memchr /home/matklad/trash/registry/src/github.com-1ecc6299db9ec823/memchr-1.0.1/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="libc"' --cfg 'feature="use_std"' -C metadata=be49c4722e8b48bf -C extra-filename=-be49c4722e8b48bf --out-dir /home/matklad/trash/fall/target/debug/deps -L dependency=/home/matklad/trash/fall/target/debug/deps --extern libc=/home/matklad/trash/fall/target/debug/deps/liblibc-90ba32719d46f457.rlib --cap-lints allow -C target-cpu=native`
Running `rustc --crate-name memchr /home/matklad/trash/registry/src/github.com-1ecc6299db9ec823/memchr-1.0.1/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="libc"' --cfg 'feature="use_std"' -C metadata=be49c4722e8b48bf -C extra-filename=-be49c4722e8b48bf --out-dir /home/matklad/trash/fall/target/debug/deps -L dependency=/home/matklad/trash/fall/target/debug/deps --extern libc=/home/matklad/trash/fall/target/debug/deps/liblibc-335251832eb2b7ec.rlib --cap-lints allow -C target-cpu=native`

Here's the single difference:

--extern libc=/home/matklad/trash/fall/target/debug/deps/liblibc-90ba32719d46f457.rlib 
--extern libc=/home/matklad/trash/fall/target/debug/deps/liblibc-335251832eb2b7ec.rlib 

Versions (whyyyyy cargo is 0.21 and rustc is 1.20??? This is soo confusing)

λ cargo --version --verbose
cargo 0.21.0 (5b4b8b2ae 2017-08-12)
release: 0.21.0
commit-hash: 5b4b8b2ae3f6a884099544ce66dbb41626110ece
commit-date: 2017-08-12

~/trash/fall master
λ rustc --version
rustc 1.20.0 (f3d6973f4 2017-08-27)

[matklad]

So, it has to do with features. Namely, two cargo invocations produce two different libcs:

Running `rustc --crate-name libc /home/matklad/trash/registry/src/github.com-1ecc6299db9ec823/libc-0.2.30/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg 'feature="use_std"' -C metadata=335251832eb2b7ec -C extra-filename=-335251832eb2b7ec --out-dir /home/matklad/trash/fall/target/debug/deps -L dependency=/home/matklad/trash/fall/target/debug/deps --cap-lints allow -C target-cpu=native`
Running `rustc --crate-name libc /home/matklad/trash/registry/src/github.com-1ecc6299db9ec823/libc-0.2.30/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=90ba32719d46f457 -C extra-filename=-90ba32719d46f457 --out-dir /home/matklad/trash/fall/target/debug/deps -L dependency=/home/matklad/trash/fall/target/debug/deps --cap-lints allow -C target-cpu=native`

The only difference is --cfg 'feature="default"'.

So, I get two different libcs in target:

λ ls target/debug/deps | grep liblibc
.rw-r--r-- 982k matklad  3 Sep 14:06 liblibc-90ba32719d46f457.rlib
.rw-r--r-- 982k matklad  3 Sep 14:03 liblibc-335251832eb2b7ec.rlib

But I get a single memchr:

λ ls target/debug/deps | grep libmemchr
.rw-r--r-- 186k matklad  3 Sep 14:09 libmemchr-be49c4722e8b48bf.rlib

The file name is the same for both cargo commands, but the actual contents differs.

[matklad]

Hm, so this looks like more serious then spurious rebuild!

Depending on what -p options you pass, you might end up with different final artifacts for the same package. This should not happen, right?

[matklad]

Minimized example here: https://github.com/matklad/workspace-vs-feaures

[matklad]

@alexcrichton continuing discussion here, instead of #4469 which is somewhat orthogonal, as you've rightly pointed out!

I don't think this'd be too hard to implement, but I'm not sure if this is what we'd want implemented per se. If one target of a workspace doesn't want a particular feature activated, wouldn't it be surprising if some other target present in a workspace far away activated the feature?

Yeah, it looks like what we ideally want here is that each final artifact gets the minimal set of features. And this should work even withing a single package: currently, activating feature in dev-dependecy will activate it for usual dependency as well. This is also something to keep in mind if we go the route of binary-only (or per-target) dependencies.

Though such fine-grained feature activation will cause more compilation work overall, so using union of featues might be a pragmatic choice, as long as we keep features additive, and it sort of makes sense, because crates in workspace share dependencies anyway. And seems better then definitely some random unrelated target activating features for you depending on the command line flags.

[alexcrichton]

I think one of the main problems right now is that we're doing feature resolution far too soon, during the crate graph resolution. Instead what we should be doing is assuming all features are activated until we actually start compiling crates. That way if you have multiple targets all requesting different sets of features they'll all get separately compiled copies with the correct set of features.

Does that make sense? Or perhaps solving a different problem?

[matklad]

Does that make sense? Or perhaps solving a different problem?

Yeah, totally, "they'll all get separately compiled copies with the correct set of features" is the perfect solution here, and it could be implemented by moving feature selection after the dependency resolution.

But I am really worried about additional work to get separately compiled copies, because it is multiplicative. Let's say you have a workspace with the following layout:

1. leaf crates A and B, which transitively depend on external crate libc with different features
2. A large number of intermediate crates, on which A and B also depend
3. An ubiquitous utils crate, that depends on libc and is a dependency of any other crate.

Because A and B require different features from libc, and because libc happens to be at the bottom of the dependency graph, that means that for cargo build --all we will compile every crate twice. Moreover, editing utils and then doing cargo build --all again recompiles everything two times.

So it's not that only libc will get duplicated, the whole graph may be duplicated in the worst case.

[nipunn1313]

If we assume that features are additive (as intended), then the innermost crate could be compiled once with the union of all features.

Additive features are a bit of a subtle point though (see #3620). Recompiling is the safest way, though expensive.

[alexcrichton]

@matklad yeah you're definitely right that the more aggressively we cache the more we end up caching :). @nipunn1313 you're also right that it should be safe for features to be unioned, but they often come with runtime or linkage implications. For example if a workspace has a no_std project and an executable, compiling both you wouldn't want to enable the standard library in the dependencies of the no_std project by accident!

I basically see this as there's a specification of what Cargo should be doing here. We've got, for example, two crates in a workspace, each which activates various sets of features in shared dependencies. Today Cargo does the "thing that caches too much" if you compile each separately (and also suffers a bug when you switch between projects it recompiles too much). Cargo also does the "union all the features" if you build both crates simultaneously (e.g. cargo build --all). Basically Cargo's not consistent!

I'd advocate that Cargo should try to stick to the "caches too much" solution as it's following the letter of the law of what you wrote down for a workspace. It also means that crates in a workspace don't need to worry too much about interfering with other crates in a workspace. Projects that run into problems of the "too much is cached" nature I'd imagine could then do the investigation to figure out what features are turned on where, and try to get each workspace member to share more dependencies by unifying the features.

[matklad]

Projects that run into problems of the "too much is cached" nature I'd imagine could then do the investigation to figure out what features are turned on where, and try to get each workspace member to share more dependencies by unifying the features.

This somewhat resolves my concern about build times, but not entirely. I am worried that it might not be easy to unify features manually, if they are turned on by private transitive dependencies. It would be possible to do by adding this private transitive dependency as an explicit and unused dependency, but this looks accidental.

But now I too lean towards fine-grained features solution.

[nipunn1313]

For what it's worth, we've done that exact trick with the parallel feature of the gcc crate. It does happen, but the workaround is ok.

…

On Wed, Sep 6, 2017 at 12:45 AM Aleksey Kladov ***@***.***> wrote: Projects that run into problems of the "too much is cached" nature I'd imagine could then do the investigation to figure out what features are turned on where, and try to get each workspace member to share more dependencies by unifying the features. This somewhat resolves my concern about build times, but not entirely. I am worried that it might not be easy to unify features manually, if they are turned on by private transitive dependencies. It would be possible to do by adding this private transitive dependency as an explicit and unused dependency, but this looks accidental. But now I too lean towards fine-grained features solution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4463 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPXoxPIsKCCcH5DEgqtKzPt9ek34uLeks5sfk2EgaJpZM4PLGrK> .

[SimonSapin]

Servo relies on the current behavior to some extent: two "top-level" crates (one executable and one C-compatible static library) depend on a shared library crate but enable different Cargo features. These features are mutually exclusive, enabling the union would not work.

Maybe the "right" thing to do here is to have separate workspaces for the different top-level things? Does it make sense for shared path dependencies to be members of two separate workspaces?

(Servo’s build system sets $CARGO_TARGET_DIR to different directories for the two top-level things so that they don’t overwrite each other. They also happen to be built with different compiler versions (some nightly v.s. some stable release).)

[nipunn1313]

I would be in support of cargo build --all building the same dependency multiple times rather than resolving a feature union. This would be equivalent of running cargo build in a loop over each crate in the workspace. This would prevent multiple crates within a workspace from interfering with each other (or multiple dependencies in the dep-chain interfering). I believe it would cover Servo's case as well.

What makes this problem so insidious is that there's no way to enforce or even encourage the union property of features. If a project pulls in even one dependency that doesn't obey this property, it could potentially create an incorrect binary.

In @SimonSapin's case with Servo, I think Servo is lucky that the feature'd crate (style) is only one-level in from the top level crate. If you had a dep chain like

evenbiggerproject -> servo -> style[featA]
                  -> geckolib -> style[featB]

then I believe that compiling evenbiggerproject with cargo would select the union of features for style and use it for both geckolib and servo. This would be an incorrect binary w.r.t. the intent of the servo/geckolib Cargo.tomls

Our project at Dropbox ran into a similar issue with itertools -> libeither, where libeither was compiled with two different features. Lucky for us, libeither's features are union-safe, so the code was correct, but it did create spurious recompiles depending on which sub-crate we were compiling.

[djc]

I agree with @nipunn1313 -- I think cargo build --all should build all crates exactly as they would be if you had run cargo build for each crate separately. If that requires us to recompile some crates, so be it.

[SimonSapin]

This all sounds like agreement on what should happen. @alexcrichton, what code changes need to happen (on a high level) to get there?

[djc]

That's what I was discussing with @alexcrichton at the RustFest impl days, and I have a bunch of refactoring done that I'm still tweaking. Will post a PR ASAP. Do you have a particular dependency/urgency relating to Gecko or Servo on this?

[SimonSapin]

Nothing urgent. I thought this bug could cause spurious rebuilds after selectively building a crate with -p, but I couldn’t reproduce. Anyway, thanks for working on this!

[nipunn1313]

We've had to fork some deps to unify feature selection to work around this issue. It's definitely not sustainable for us, but not urgent yet.

…

--Nipunn

On Sat, Oct 14, 2017 at 5:27 AM Simon Sapin ***@***.***> wrote: Nothing urgent. I thought this bug could cause spurious rebuilds after selectively building a crate with -p, but I couldn’t reproduce. Anyway, thanks for working on this! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4463 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABPXo7HL7OuZhSaMgZMtY9Y5IxnY7dHFks5ssKjIgaJpZM4PLGrK> .

[djc]

@nipunn1313 for my understanding, can you point me at a commit or otherwise elaborate on what problems you've had due to this issue?

[nipunn1313]

Here's an example of a problem we had to work around
#3620 (comment)

In that particular case, either and itertools were both present in our workspace.
We ended up internally forking itertools to ask for a wider set of features from libeither, so there was consistency across the workspace.

[alexcrichton]

@SimonSapin taking on this issue will require a relatively significant refactoring of Cargo's backend. Right now feature resolution happens during crate graph resolution, but we need to defer it all the way until the very end when we're actually compiling crates.

[driftluo]

Today, I encountered a problem with the overall compilation of the workspace and the inconsistent compilation of each crate.

We have a public logger crate, the default is output to a file, open console features will be output to the terminal.

Some crates in the workspace have this feature turned on, while others do not need to be turned on.

But cargo bulid --all will cause the entire workspace's crates to be output to the terminal.

This behavior makes me very confused.

[idubrov]

Regarding:

Projects that run into problems of the "too much is cached" nature I'd imagine could then do the investigation to figure out what features are turned on where, and try to get each workspace member to share more dependencies by unifying the features.

1. What would be the best way to figure out which features needs to be unified?

I hacked together a crude tool that compares output of multiple cargo build --build-plan -Z unstable-options --package XYZ runs to see if compiler would be invoked with a different set of features for the same version of dependency for different packages. Is there a better way to do it?

2. Would it be feasible to ask for a feature to "lock" features on the workspace level, similar how versions are locked via Cargo.lock? Something along the lines (in the workspace Cargo.toml):

[enable-features] # some better name
# for all the dependencies of `syn` in the workspace matching version `0.15`,
# forcefully enable "visit-mut" and "extra-traits" features.
syn = { version = "0.15", features = ["visit-mut", "extra-traits"] }

?

This is pretty much to avoid crude solution I outlined here: https://users.rust-lang.org/t/forcing-set-of-features-for-shared-dependencies-across-targets-in-workspace/25899 (I don't really want to have "fake" dependencies or tune features across multiple Cargo.toml files).

3. So far I found two use-cases for feature unification:

First, it improves compilation times (as explained in my post on users.rust-lang.org).

Second, I found that inconsistent features break the following use-case I have (which might be a not fully valid one?):

1. shared crate is compiled as dylib
2. plugin crate is compiled with shared as a dependency; plugin itself is dylib
3. app crate is compiled as a binary with shared as a dependency; NO direct dependency on plugin

app tries to load plugin at runtime via libloading library.

So, when features are consistent on shared between two targets (plugin and app), everything works fine. I can build plugin and app independently, start app and it will load plugin. Code is shared between these two in the form of libshared.dylib dynamic library.

However, when features are not consistent (say, plugin=>shared=>another_dep enables some feature A in another_dep and app=>shared=>another_dep does not), this causes plugin to fail loading because some symbols would be missing in libshared.dylib. Also, it seems like a single version of libshared.dylib is created for both targets, so building app & plugin independently would constantly override that libshared.dylib.

It seems like dynamic libraries is still an unfinished story in Rust, so I don't know what really I could be asking here (it might be that it "works" only accidentally?).