Cargo is unable to resolve dependency versions

[weiznich]

Problem
Cargo is unable to select two versions of diesel as dependency of some crate.

error: failed to select a version for `diesel`.
    ... required by package `infer_schema_internals v1.3.0`
    ... which is depended on by `infer_schema_macros v1.3.0`
    ... which is depended on by `diesel_infer_schema v1.3.0`
    ... which is depended on by `diesel_tests v0.1.0 (/home/user/Docs/rust/diesel/diesel_tests)`
versions that meet the requirements `~1.3.0` are: 1.3.3, 1.3.2, 1.3.0

all possible versions conflict with previously selected packages.

  previously selected package `diesel v1.4.0`
    ... which is depended on by `diesel_demo_step_1_pg v0.1.0 (/home/user/Docs/rust/diesel/examples/postgres/getting_started_step_1)`

failed to select a version for `diesel` which could resolve this conflict

I've seen this working if the other version of diesel is 0.x, so this does not make any sense for me that a it is not possible to use two semver compatible versions at the same time as dependency if explicitly requested.

Detailed dependency situation:
We have a workspace containing several crates. Most of them depend on diesel = "1.4.0". Additionally there is one crate (diesel_tests), that depends on diesel_infer_schema = "1.3.0". Internally diesel_infer_schema depends on diesel = "~1.3.0" which causes the version selection problem shown above.

Steps

1. Clone diesel
2. Change dir to /path/to/diesel_clone/diesel/
3. Try to build diesel

Possible Solution(s)
Cargo should be able to use 2 versions of the same dependency if requested explicitly even if there would be an semver compatible version if the versions are specified in a other way.
(I.e. That should just work)

Notes

Output of cargo version:

• cargo version: cargo 1.31.0 (339d9f9c8 2018-11-16)
• rust: 1.31.0 (installed through rustup)
• OS: ubuntu 18.10

cc @sgrif @killercup @Eijebong (That breaks our CI)

[sgrif]

One other note is that it had no issues resolving this with two different versions when the version numbers were the same. Prior to the release of 1.4, we had diesel 1.3.3 (from crates.io) and the local version of diesel (also versioned 1.3.3 but with different code) both in the same build. The crates.io version was used as a dependency for diesel_infer_schema, while the local version was used for everything else.

[alexcrichton]

The version requirements 1.4.0 (implicitly ^1.4.0) and ~1.3.0 are incompatible because Cargo can't select one unique major version in the 1.x.y series, but perhaps a different requirement was intended somewhere?

[weiznich]

Both requirements are intended as written there. diesel_infer_schema is unmaintained an will not get any new updates, but it uses some API considered as internal to diesel. Therefore we cannot allow it to use newer version of diesel.

@sgrif It worked before because cargo was able to select a version (1.3.3) for both. That [replace] in the root Cargo.toml then pointed diesel 1.3.3 to that local version in our repo.

[alexcrichton]

Ok! In that case I think this is not a cargo issue, right?

[weiznich]

Let's elaborate this in another way: I do not understand the reasoning behind that version resolution schema. Yes cargo should try to select one common major version, but if that's not possible (because one crate uses a explicit version as internal dependency for example) it should be possible for cargo to just select 2 version. Additionally that already works fine if the dependencies have different major versions.
(I just cannot come up with a reason why this may be a bad idea.)

[sgrif]

@alexcrichton Cargo should allow two different versions of this library to be used, shouldn't it? My understanding was that incompatible version constraints didn't matter here, it'd just use 1.3 for the library that required it, and 1.4 where that was requested. It was previously using two versions before this, they just happened to have the same version number.

Looking in the Cargo.lock for crates.io, at the very top I see that we have aho-corasick 0.5.3 and 0.6.3. It doesn't seem like Cargo should be considering these two incompatible constraints against each other at all, it should just be using two different versions of the crate.

[sgrif]

Actually looking at Cargo.lock for crates.io, it even has two different versions of Diesel there. 0.16 and 1.3. The only difference here is that the bounds don't use ~. My understanding was that ~ was just a more strict requirement than ^, not that it'd try to force it to be resolved to a single version across the whole tree.

[alexcrichton]

@weiznich

Let's elaborate this in another way: I do not understand the reasoning behind that version resolution schema

Ah that's just how Cargo has always worked. Cargo allows multiple major versions of a crate, but all selected versions of a crate must be semver-incompatible (e.g. can't have 1.3.0 and 1.4.0 as they're semver compatible).

There's a whole bunch of reasoning behind why Cargo does this, but it's probably a bit much to get in here. We'd like to change this, we just haven't yet.

I just cannot come up with a reason why this may be a bad idea

One example is that if you're using 1.3.0 and I'm using 1.4.0 and someone tries to use us both, they get compile errors that look obscure as opposed to a version resolution error. Instead we need to be using the same version to be used together.

---

@sgrif

Cargo should allow two different versions of this library to be used, shouldn't it?

Yes, but only if they're semver incompatible (and 1.3.0 and 1.4.0 are compatible, 1.4.0 wasn't a breaking change relative to 1.3.0 semver-wise)

Looking in the Cargo.lock for crates.io, at the very top I see that we have aho-corasick 0.5.3 and 0.6.3.

That's because Cargo's interpretation of semver means that 0.5.x and 0.6.y are semver-incompatible. The 0.6 branch was a breaking change relative to the 0.5 branch.

The ~ requirement is indeed more strict than ^, but Cargo strongly discourages usage of any version requirement other than ^ unless you're really sure you know what you're doing

[weiznich]

One example is that if you're using 1.3.0 and I'm using 1.4.0 and someone tries to use us both, they get compile errors that look obscure as opposed to a version resolution error. Instead we need to be using the same version to be used together.

The same already happens if you try to use 2 semver incompatible versions, so that would be nothing new. (And that only affects dependencies exposed in public interfaces). So that seems to be a weak argument for me.

The ~ requirement is indeed more strict than ^, but Cargo strongly discourages usage of any version requirement other than ^ unless you're really sure you know what you're doing

That's the actual point here: We "know" what we are doing. That one version is a internal (build-)dependency. That other one a "public" (runtime) dependency. So they cannot collide…

[sgrif]

Yeah, the semver incompatible restriction seems really pointless to me since you get those exact same errors regardless of semver compatibility. Would this need an RFC to be changed?

[alexcrichton]

This isn't really the place to debate the merits of Cargo's current implementation. Since Cargo is working as intended I'm going to close this issue, but a follow-up can be opened which is targeted about possibly changing Cargo's behavior.

For changing this, yes it would require an RFC

[sgrif]

Thanks for taking the time to clarify this for us

[sgrif]

@alexcrichton I actually think there still is a bug here, since Cargo will allow multiple versions of a dependency if they have identical version numbers (when one is depended on as crate = "1.0.0" and one is depended on with path = "local_path"). Shouldn't this be disallowed since they are semver compatible?

[alexcrichton]

Cargo considers those two different crates because they come from two different sources. One is from crates.io and one is from a "path source". Conflicting versions are allowed from different sources because the different sources produce different crates that can't ever be unified.

[kevinji]

@alexcrichton Could you provide some more color about the reasons why Cargo doesn't support multiple semver-compatible versions of the same package? I've been running into this more when crate authors disagree about semver-compatibility over MSRV bumps that causes a lot of painful upstream changes; personally, I've needed to fork a lot of packages just to tweak Cargo.toml bounds which is difficult to maintain.

If there's support for this change, I would like to help draft an RFC so any guidance around that would also be helpful.

[Eh2406]

I don't think a closed issue from three years ago is likely to be a good home for this discussion.

I do not have inside information, I was not around when this compromise was made. But as a current member of the cargo team, one of the places I've recently written up my understanding is:
https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/Cargo.20and.20duplicating.20dependencies

If the cause of the arguments over versioning is MSRV tensions, then it is likely to be more productive to suggest changes that directly affect MSRV aware resolution.

Any RFC that allows multiple semver compatible versions will need to describe precisely when it will be allowed, when it will not be allowed, how the resolver will decide if it is allowed, how that decision will be made clear to the user who cares, and how a user would be able to control if it applies on a case-by-case basis. Resolution is already NP-complete so none of those questions are straightforward.