Enable yanked dependencies with an unstable flag

[pietroalbini]

This PR adds a new unstable CLI flag, -Z allow-yanked-deps, that allows to depend on yanked crates even if there is no existing lockfile pointing at them (it basically disables the yanked check).

Crater needs this because it regenerates lockfiles on every run, so there are tons of crates that fail to build because they depend on yanked deps (like futures 0.2). Crater already uses unstable flags during lockfile generation, so I don't really care if this is stabilized or not.

Closes #4225
cc rust-lang/crater#243

[rust-highfive]

r? @matklad

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

Thanks for the PR! The implementation and idea here are fine by me, but I'm wary to merge perma-unstable features. We've had a very bad track record of features in rustc which are merged but have no path to stabilization, and I would prefer to avoid the same in Cargo.

To that end, do others on @rust-lang/cargo perhaps see this flag ever becoming stable? Do we think we could ever stabilize functionality like this?

[joshtriplett]

I don't think we should have a stable way to do this, and if Crater didn't need it I'd argue we shouldn't have an unstable way to do this either.

[Eh2406]

Several of our unstable functionality are because Crater needs it. In addition to internal implementation detail, and on the path to stabilization, we may need a intended for rust-lang tooling endgame for features.

[matklad]

Several of our unstable functionality are because Crater needs it.

I suggest delisting such features from docs and --help message perhaps?

[alexcrichton]

@pietroalbini to get a sense of scale here, how many crates fail to compile due to a dependence on yanked crates? What percentage of all crater-tested crates does this cover?

[alexcrichton]

@Eh2406 do we have other unstable functionality specifically for crater? One possible one, -Z offline, is something I believe we'd like to stabilize for Cargo itself at some point

[pietroalbini]

to get a sense of scale here, how many crates fail to compile due to a dependence on yanked crates? What percentage of all crater-tested crates does this cover?

At the moment ~170, which is 1% of the crates we're testing. The other option we have right now is to manually blacklist them, or have them sit in the "Crater errors" section of the report (which is not ideal, I'd like to have nothing in it so we can catch new ones easily).

do we have other unstable functionality specifically for crater? One possible one, -Z offline, is something I believe we'd like to stabilize for Cargo itself at some point

Crater doesn't use -Z offline because that flag alters dependency resolution. Instead it uses -Z no-index-update (added in #4990), which just assumes the registry is up to date and skips updating it.

[Eh2406]

do we have other unstable functionality specifically for crater?

when I said it it looked like #5961 was going to be in the list as well.

[alexcrichton]

Ah sorry I thought I replied sooner. @pietroalbini 1% seems like a pretty small number and given the historical resistance (and current resistance) to this feature, would you be ok adding such crates to the blacklist?

[bors]

☔ The latest upstream changes (presumably #5995) made this pull request unmergeable. Please resolve the merge conflicts.

[pietroalbini]

I'd prefer to wait on this until I import all the GitHub repositories into Crater, to see if the numbers change.

[pietroalbini]

Ok, the number of yanked crates didn't really change. Closing this.