New behavior of `--feature` + `--package` combination

[matklad]

A tracking issue for #5353 and #5351. Historically, --feature and related flags applied to the current package, and not to the package, specified via --package. This is a bug, but fixing it could break someone's code, so currently new behavior exists under -Z package-features opt-in. We need to check if it breaks code in the wild to see what we should do next.

[matklad]

#5362 is I think the manifestation of the old odd behavior.

[matklad]

Announcement on irlo: https://internals.rust-lang.org/t/help-us-test-the-breaking-bug-fix-to-cargo-features/7317

[matklad]

I think #4942 is also this issue.

[matklad]

And #4753 as well

[dwijnand]

(typo in the issue description: --pacakge)

[matklad]

This broke Servo unfortunately, so we probably would need to back this out, at least in this, maximally aggressive, form.

However, I wonder what we plan to do with feature-selection problems in general?

I assume that we want to eventually fix the main problem with features. Currently, features are global per compilation, and that means that features bleed from dev-deps to normal deps and that cargo build -p foo and cargo build -p foo -p bar build the foo package differently.

Fixing that, however, would be a breaking change: suddenly, Cargo would select a smaller set of features :-(

But, if we do fix that, then behavior of the combination of --package and --features would necessary change as well. In other words, this PR is sort of a subset of that larger issue.

The core question I think is do we feel comfortable with eventually changing the set of activated features? If we are, then I think we can modify current fix to not flatly error-out, but to ignore the features argument for several packages, printing a warning like "this was accepted in previous versions of Cargo and activated features in surprising way, and not it is ignored".

[pravic]

$ cargo +stable test --all --features xyz

Used to work. And expectation was "Enable the specified set of futures for crates that support it".

$ cargo +nightly test --all --features xyz
error: cannot specify features for more than one package

Now we see this (thanks to CI to catch it soon and not after Rust release) and it does not matter whether all of workspace members have xyz or not, error is the same.

How come that the original issue (cargo --package foo --feature feat) has affected builds without --package? #5390 wasn't a good idea without a compatibility explanation (what is it? how to fix that mystical error? how it will work in future?). That error must have contained at least the issue number (since it is a breaking future and it is in nightly (i.e. unstable) anyway).

[matklad]

And expectation was "Enable the specified set of futures for crates that support it".

Not that this expectation is wrong. What actually happens is that the feature is enabled only for the package in the current working directory. In other words, it is irrelevant whether all members have the xyz feature, because this applies only to a single member, and not the one that is specified via the -p argument.

That error must have contained at least the issue number

That was definitely an oversight on my part :(

[matklad]

Here's a small example which demonstrates the difference in behavior: https://github.com/matklad/features-behavior

[matklad]

has affected builds without --package?

--all is a shorthand for --package member1 --package member2 --package member3 etc, and suffers the same problem: --feature flag does not apply to the selected packages. I've extended the example at https://github.com/matklad/features-behavior to show how this affects --all.

[SimonSapin]

Breaking changes to stable functionality are not acceptable. Please revert and find a way to make this opt-in.

However weird and subjectively broken the old behavior is, it has been around for more than long enough that many large projects have built work-arounds that came to rely on it. It’s easy to say this in hindsight, but I think it shouldn’t be surprising that a change like this is definitely gonna break some builds.

#5390, the PR that enable this change by default, says:

So far, the feedback on https://internals.rust-lang.org/t/help-us-test-the-breaking-bug-fix-to-cargo-features/7317 has been positive

At the time of that writing, that forum thread was 2 days old and had responses from three individuals, one of them saying their build is broken. How is this "positive" enough?

But even if that discussion had received community-wide attention, breakage like this without opt-in is still unacceptable IMO. The projects most likely to be affected are those with complex build system that involve many crates. They may or may not be involved in the day-to-day development of Rust and Cargo.

---

Now if we are going to make a change around feature selection, in addition to being opt-in (possibly through edition = "2018" in the root crate or workspace?) I think we should spent a lot more time and involve more people to consider what the new behavior should be. In particular, fixing #4463. And possibly separating the notions of “Choosing which of the several 'root' artifacts in a workspace (each with its dependency graph and feature selection)” and “Choosing a subset of one dependency graph”.

[gnzlbg]

Because of this cargo bug stdsimd has been silently not testing some feature combinations (rust-lang-nursery/stdsimd#569). That this produces no warnings whatsoever worries me.

[gnzlbg]

Breaking changes to stable functionality are not acceptable.

This is incorrect. Breaking changes to stable functionality are acceptable if the functionality is broken, otherwise we couldn't ever fix any bugs if the behavior of the fix could be observed by current crates.

The question that we have to resolve here is whether this functionality is broken or not. If it isn't broken, you are correct in that we cannot fix it. But if it is broken, we can fix it. Depending on the impact of the fix, we might have to warn first for a sufficiently long time before applying the fix, but fixing it is possible and does not require a new edition, although we could use one to implement the fix.

Currently, when I have a workspace with a package foo with a feature bar, executing:

cargo test --features=bar -p foo

compiles foo without the feature bar. I personally think this behavior is broken, and that those wanting this behavior should just write cargo test -p foo without --features=bar.

[SimonSapin]

We can split hairs about the finer details what the stability promise means, but in this particular case it is clear that a number of projects have been negatively impacted by this change until it was reverted, with no clear migration path.

warn first for a sufficiently long time

I don’t think that’s good enough. A change of this importance needs to be opt-in, either with a dedicated line in the workspace’s manifest or as part of an edition switch.

[gnzlbg]

but in this particular case it is clear that a number of projects have been negatively impacted

After going through this and related issues, including the servo one, the only thing that's clear to me is that this change has broken the build of some projects. The fact that these projects have been negatively impacted by this change is not at all clear to me, since I haven't find much information about why exactly the build broke for those projects and whether the previous behavior was even actually intended.

FWIW this change would have broken the build of stdsimd too, and in doing so it would have discovered a bug in stdsimd's CI that we would have long fixed. Therefore, stdsimd would have been positively impacted by this change, even if the change would have required some work on our part to fix.

[SimonSapin]

Just to not get things mixed up:

• The “new behavior” mentioned in this issue’s title is the one enabled in #5353 and reverted in #5430.
• The “current behavior” is the one that has existed for as long as workspaces and feature flags have both existed, and is the same in today’s Stable and Nightly.

---

Breaking the build in a way unrelated to unstable features is negative impact. I don’t remember the details at this point, but we had found work-arounds for the current behavior’s weirdness, and in applying those work-arounds had come to rely on the current behavior.

The new behavior was present in Nightly for about two weeks (between rust-lang/rust#49939 and rust-lang/rust#50344). I don’t know why the stdsimd bug was not discovered during that time. For Servo we have a daily Travis-CI job that overrides the pinned rust-toolchain file in the repo and tries to build with the latest Nightly, notifying a few people if that fails.

[dhardy]

I don’t think that’s good enough. A change of this importance needs to be opt-in

I don't think the status quo is good enough either; this bug should have been fixed a long time ago. But I agree that the #5353 may not be the ideal solution.

error: cannot specify features for more than one package

--feature flag affects the selected package. It is an error to
specify --feature with more than a single -p, or with -p outside
workspace (the latter could work in theory, but would be hard to
implement).

This is obviously a problem, because people have used expressions like cargo test --all --features foo repeatedly. But that in itself is a problem since the expression doesn't do what many expect.

And expectation was "Enable the specified set of futures for crates that support it".

Not that this expectation is wrong. What actually happens is that ...

We could implement this expectation. On the positive side, it should only break tests if those tests were in fact testing broken code. On the negative side, a typo or forgetting to add a feature to a sub-package could go unnoticed.

But what might be a reasonable compromise would be to enable features specified by --feature for all selected packages, but also add an --allow-missing-feature option which silently ignores features on packages without them.

---

Summary: a proper fix probably will cause some breakage to existing build systems. But it should correct mis-conceptions and should be very easy for users to fix.

IMHO it is worth suffering some breakage here. Note also that the breakage will mostly only affect continuous integration testing, and not users building/testing packages manually and depending on them transitively.

[SimonSapin]

a proper fix probably will cause some breakage […] IMHO it is worth suffering some breakage here.

Breakage and fix v.s. neither are not the only options! We can have a configuration in the workspace’s root Cargo.toml to make the behavior change opt-in.

Note also that the breakage will mostly only affect continuous integration testing, and not users building/testing packages manually and depending on them transitively.

IIRC this was not the case with Servo, we could build at all. I’m not sure why you make this distinction by the way, CI is designed to be as close as possible to other usage.

[gnzlbg]

We are at an impasse here.

If we make this change opt-in, we don't break any code in the wild. This is good because it does not break code in the wild that's correctly working as intended. At the same time, this is bad because there is code in the wild that's silently broken, and this change does neither fix it nor alert the maintainers of this code that it needs fixing. If we make this opt-out, we break the broken code, which is good, but we also break the correct code, which is bad.

Honestly, we are here because the current behavior is too implicit, different people expected it to behave differently, and things just silently happened to work for everybody, resulting both in correct and incorrect code.

A compromise solution to this problem might be to just fix the implicitness. We could just add two flags, that enable the different behaviors, and just start warning that all projects should explicitly specify which behavior they want somehow (how exactly is something we can decide later). That will be better than what we currently have, it does not break any code, and it does let people with broken code know that their code might be broken.

Whether we ever turn this warning into an error or not, and how we would do that, is the most controversial part of this issue, and something that we don't really have to discuss right now.

[dhardy]

Then I assumed wrong about Servo... why are you using --package with multiple packages for any other reason than testing? Is this easy to change to mitigate the breakage caused by #5353? I tried following the links above but didn't find much information.

We can have a configuration in the workspace’s root Cargo.toml to make the behavior change opt-in.

IMO this is an ugly solution — many users will forget or not know about this, then be surprised when down the road they find thing doesn't work with feature foo despite cargo test --package thing --feature foo being in their CI.

Cargo's configuration and usage is already complex with significant documentation users need to read to understand various features. Adding further configuration options should not be a preferred option IMO.

A compromise solution ... We could just add two flags

At least this doesn't implicitly do the wrong thing for anyone, and if it only shows up when using --feature with --package / --all then it doesn't affect everyone else. Acceptable, but I'd still prefer that this is just "fixed" for everyone (with easy solutions for those affected).

[arcnmx]

Okay it's incredibly frustrating that combining --package with --features is completely broken and does not produce any warnings or errors at all. Anyone assuming that cargo implements the reasonable expected behaviour here will see CI feature tests pass without issue, having no indication that crates are simply being built without the features enabled.

It's frustrating to see that there was a fix made and then suddenly reverted, with no other mitigation or progress since. This seems to be a pretty obvious footgun where most cases seem to be people using --all or -p X and not realising the interactions/implications, with parts of code silently just not being built or tested. It happens to work when a root crate has feature = ["child_crate/feature"]. With no root crate, using --features should probably error to indicate it's broken (#5849) or otherwise just use the fixed behaviour.

Breaking the build in a way unrelated to unstable features is negative impact. I don’t remember the details at this point, but we had found work-arounds for the current behavior’s weirdness, and in applying those work-arounds had come to rely on the current behavior.

Relying on obviously broken and weird behaviour, considering it to be a workaround in build scripts, not using other workarounds that rely on more established reasonable behaviour (like changing CWD rather than using -p/--all), and never expecting it to be fixed seems kind of unreasonable? There are solutions that could be employed to cause less breakage or even be partially or fully compatible with the old behaviour, but it seems unclear from this issue and the surrounding discussion why or how the old behaviour was relied on in the first place.

[alexcrichton]

@arcnmx breaking changes, whether they're fixing bugs or not, take care when approaching. We can't apply this change and then say "y'all deal with the fallout, it's a change for the better anyway". Our policy is that we would slowly roll out this change, providing warnings and such along the way. For example we would need to:

• Add a configuration switch to enable the new beahvior
• Add warnings today when the new behavior and the current behavior differ
• Push to the ecosystem, evaluate fallout
• Eventually, when ready, change the defaults.

This takes a lot of work and no one has been sufficiently motivated to drive it to completion.

[sfackler]

Apologies if this has already been covered upthread, but would it be possible to make this change for virtual workspaces specifically? In that case, there is no current package, so it seems less likely to break the world.

[arcnmx]

Yeah, my intention is to revive the discussion here and determine what intermediate steps and migration plans are possible, it's just quite frustrating that the proper behaviour is inaccessbily stuck gated behind an unstable/nightly flag. My main points of concern are:

1. Considering that virtual workspaces are completely broken to my knowledge when combined with feature flags, straight-up enabling the expected/fixed behaviour there would be a good way to address #5849. Otherwise it really should error outright to prevent confusion by the current false assurance that everything is just fine.
   • as long as there's agreement that the new behaviour is correct (considering the original PR was approved, presumably is)
2. Identifying why the change broke things in the first place, as I'm unable to find explanations of how and why the broken behaviour is currently relied on by existing projects. Most of the outcry doesn't seem to articulate why the revert was necessary, and...
   • whether there's a more conservative or backwards-compatible approach that could be used here?
   • otherwise / irrespective of the above going ahead with such a config/flag warning cycle plan is quite reasonable, with some agreement on a rough approach to follow

[dwijnand]

Perhaps switching the defaults (i.e hard fail rather than just warn) could also have an "opt-back" option (mentioned in the error message) for builds that need a second before migrating. Then we can drop the opt-back option.

[alexcrichton]

Given the fallout we previously experienced I'd be wary of changing this even for virtual workspaces myself. I'd personally prefer that we take a conservative route of "warn if things are different, allow configuration for new, 'correct', behavior"

[ririsoft]

Hi, basic developer using rust and getting into this issue here. I agree with the proposal to change the behavior and I understand the concerns regarding stability of the current yet buggy behavior.

Given the fallout we previously experienced I'd be wary of changing this even for virtual workspaces myself. I'd personally prefer that we take a conservative route of "warn if things are different, allow configuration for new, 'correct', behavior"

This proposal would be perfectly fine for me: When doing cargo build -p foo --feature bar at workspace level I should get a warning telling me that I am using a feature at workspace level which is not what I might expect. It should suggest to use a .cargo/config or command line option to opt-in into the natural "fixed" behavior.

Cheers.

[dhardy]

Alternative solution: deprecate the --package and --all options. To be clear: I don't consider this a good solution, but if the combination of --feature and --package is not fixable, this is an alternative.

We already have another way of testing sub-packages:

cargo test --manifest-path rand_core/Cargo.toml --no-default-features --features=alloc

[cbeck88]

@dhardy Thanks very much for pointing out this effective and non-obvious workaround for this issue!

[Nemo157]

Because of #5932 that workaround won't work in all cases, but it should work for any workspace that doesn't use default-members.