override requirements in conan install

[wpalfi]

In the docs you explain requirement overriding. I guess that can be used to handle diamond dependency conflicts. You force existing packages to use different versions of their requirements.

But what if a library needs to be rebuilt to correctly use different versions of its requirements? We would need to create different packages of that library. We can handle that by adding an option and using conditional requirements in the conanfile. Wouldn't it be nice to be able to override the requirements the same way you override options and settings in the conan install command? Someting like:

conan install . --requires="LibA/0.1@demo/testing"

[memsharded]

Yes, this could make sense, I have thought about it a couple of times, specially when doing CI.
We should take into account how to specify when they are "overrides". Or maybe they are always overriding?

[lasote]

Just to recall, @memsharded. Probably it has to be analyzed together with the idea of the conaninfo becoming a profile, requirements overriding in a profile (that would be the same as the proposed --requires) narrow even more the difference between them. That means, in CI => reproduce a build knowing the package id (having the conan_info, so, having a "profile"?).

About your question about if they should be always "overrides" or not, maybe the answer is also in the conaninfo?

[wpalfi]

We could do:

options = {"requireA": "ANY", "requireB": "ANY"}
default_options = "requireA=LibA/0.1@demo/testing", "requireB=LibB/0.1@demo/testing"

def requirements(self):
    self.requires(self.options.requireA)
    self.requires(self.options.requireB)

Do you think that is a good idea?

[mpdelbuono]

That is very similar to what I am doing now, though more accurately I'm only including the channel part because that's all I need for my case.

I ran into this problem due to the need to simultaneously build multiple branches, which we mapped to channels, but not being able to tell which branch to pull requirements from. At first we were using scopes to handle this but (1) scopes are being removed and (2) there is some level of danger in not encoding the override information into the package.

Having a clean solution for this would be nice. It seems like it would not be uncommon for multiple parallel branches of development to be occurring for larger projects.

[memsharded]

@mpdelbuono but for the user/channel, you might probably be good with the self.user and self.channel attributes, that allow to propagate user/channel all upstream the hiearchy. Are you aware of those? Won't fit your use case?

there is some level of danger in not encoding the override information into the package.
The override information might not be reflected on the package at all, it depends on the package_id() details, but the idea is that if it affects some levels upstream, it probably shouldn't affect the binary.

Thanks for the feedback!

@wpalfi The thing about that approach is that it doesn't allow to really override upstream transitive dependencies only your current project one. Many times you have A->B->C->D. You are installing A, and want to depend on a new specific version of D (D.2), instead of the D.1 that C requires. You don't want to implement that in all packages, then you need to know which packages actually depend on D. So the idea would be to add overriding requirements to A, that will be propagated upstream, even if they are not modeled as direct A dependencies.

[mpdelbuono]

@mpdelbuono but for the user/channel, you might probably be good with the self.user and self.channel attributes, that allow to propagate user/channel all upstream the hiearchy. Are you aware of those? Won't fit your use case?

Unfortunately not, because all of these dependencies live in different repositories by potentially completely disconnected teams. As such, I can't just propagate the channel up the hierarchy because there is no defined relationship between channel names.

For example, one team might start working on a feature that they don't feel is stable enough for the development channel. So they create a separate channel for it. That new channel has no meaning in any of the other packages, and so a different channel must be selected. Similarly, for consumers that wish to take an early version of that feature, they will need to select that channel, but will need to continue selecting the main development channel for other packages.

A little weird to describe; I hope that made sense.

[vctgross]

Hello folks, sorry for being late to the party, but there is something I don't quite understand in the original question:

But what if a library needs to be rebuilt to correctly use different versions of its requirements? We would need to create different packages of that library.

Is this user story not already covered by self.info.requires.full_version_mode() in package_id() ?

Consider this toy example (everything is in user/channel foo/bar)

class A1Pkg(ConanFile):
    name = "aaa"
    version = "1.0"
    requires = "bbb/1.0@foo/bar", "ccc/1.1@foo/bar"

class A2Pkg(ConanFile):
    name = "aaa"
    version = "2.0"
    requires = "bbb/1.0@foo/bar", "ccc@1.2@foo/bar"

class BPkg(ConanFile):
    name = "bbb"
    version = "1.0"
    requires = "ccc/1.1@foo/bar"

    def package_id(self):
        self.info.requires.full_version_mode()

class C1Pkg(ConanFile):
    name = "aaa"
    version = "1.1"

class C2Pkg(ConanFile):
    name = "ccc"
    version = "1.2"

$ conan install aaa/1.0@foo/bar --update
aaa/1.0@foo/bar: Installing package
Requirements
    aaa/1.0@foo/bar from local cache
    bbb/1.0@foo/bar from local cache
    ccc/1.1@foo/bar from local cache
Packages
    aaa/1.0@foo/bar:c668b935c26444bdbf9523b01fdd5c7c41b8d50e
    bbb/1.0@foo/bar:42a5db51d66d1ca71a84de56f3923e13ba6ac8f7
    ccc/1.1@foo/bar:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9

ccc/1.1@foo/bar: Already installed!
bbb/1.0@foo/bar: Already installed!
aaa/1.0@foo/bar: Already installed!

$ conan install aaa/2.0@foo/bar --update
bbb/1.0@foo/bar requirement ccc/1.1@foo/bar overriden by aaa/2.0@foo/bar to ccc/1.2@foo/bar 
aaa/2.0@foo/bar: Installing package
Requirements
    aaa/2.0@foo/bar from local cache
    bbb/1.0@foo/bar from local cache
    ccc/1.2@foo/bar from local cache
Packages
    aaa/2.0@foo/bar:c668b935c26444bdbf9523b01fdd5c7c41b8d50e
    bbb/1.0@foo/bar:b01ca7d831a5746c27ea9695d16a8429a4799eee
    ccc/1.2@foo/bar:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9

ccc/1.2@foo/bar: Already installed!
bbb/1.0@foo/bar: Already installed!
aaa/2.0@foo/bar: Already installed!

As you can see (and hopefully reproduce) bbb package id is different depending on which ccc version is used.

Of course, it relies on the fact that aaa overrides bbb idea's of ccc version, but you could have multiple conanfile.txt, or multiple branches each with a specific conanfile.txt:

[requires]
    bbb/1.0@foo/bar
    ccc/1.2@foo/bar

[requires]
    bbb/1.0@foo/bar
    ccc/1.1@foo/bar

Sorry if I am stating the obvious.

[memsharded]

@vctgross

Not really. The package_id() controls the binary packages that are created from a recipe. Even if you have a package_id() that has a full_version_mode() enabled, the problem is the following:

• You have B1.0->C1.0
• Now you have A->(B1.0, C2.0), so C2.0 is required, and a different binary for B1.0 is necessary.
• How can you fire a conan create from the B repository, so you force to consume and link with C2.0 without changing the B recipe? Because you still want B1.0 to use C1.0 by default.

In that case you need to somewhat define that "override" of C2.0 when you are building B1.0 directly from B1.0 (conan create), not as a dependency of A (conan install --build=B).

[gsemet]

I would a feature just as the proposed

conan install . --requires="LibA/0.1@demo/testing"

This shall mean "If at any point you see a requires for LibA, just take this version without trying to respect all other restrictions user/channel or version". I had to develop my own base class that take this information in the current lib + all other dependencies that are built and force the version under test. This is a bit too much invasive and if any of the dependency does not use my base class, the overrides does not work. Would it be possible to add such a feature?

[lasote]

We have been discussing this issue lately but we need more information about your use cases if you don't mind.
I see a lot of upvotes to the original comment but we are still lacking some motivation to implement this, which, at first sight, looks a bit weird because typically you want your consumer to declare explicitly the requirements. Are you requesting this for CI to try something? or just locally while trying something?... or... we need a bit more context.

Edit: I see some feedback from this duplicated issue #5067 but I would like to hear more from here.

Edit2: Note that some of the use cases described there, might be related to version ranges and could be solved using the lockfiles feature

[Minimonium]

We'd appreciate the feature because it'd allow us to try out "experimental" packages locally which do not conform to the version-range protocol without changing the recipe. We had cases where after local changes - a developer forgot to fix the dependency back from an experimental and it was leaked into the trunk, which is while not that crucial but very inconvenient.

For us, It's purely a development feature.

[gsemet]

Hi. For myself, I have developed a tool and its companion conanfile base class that allows me to inject these dependencies at run time. the main use case for me is is to test a change before merge:

• libA is used in libB.
• libB is used in appA
• a change on libA requires a new version of libB, and both should be tested in libA.
• If something breaks on libB or appA, merge libA is blocked.
• If a change is required on libB or appB, one or several merge request needs to be linked to libA. (for the moment we rely on the same git branch name, publishing under a predefined user/channel with git branch name in it to allow several such test in parallel)
• when libA is merged, a new version or libA can or might not be released, but changes on libB and appA will only be integrated upon dependency (lockfile) updates. Updates on libB or appB are done when desired.
• optionally, all merges can occurs at the same time when lockfile are not used (libB would not use lockfile, while appA would).

I know this is not "normal" use case for a clean, independent library that would carry its own intensive unit test campaign. It is not like that I write my python packages for instance. Here I have used "lib" and "app" but it is just to show the relationship between them.

For some statistic, so far we have a few dozen of such packages, and we might increase rapidly to 100 and maybe more. Using conan is extremely useful to avoid rebuilding from scratch each time (it is C/Cmake underneath), but in these 100 packages, a change on one can impact some other, so we need to check on the integration project if something break as soon as possible, to warn the developer and also to help it identify what other change will be required:

• a function is renamed on libA (breaking change, major update)
• libB needs a patch to use this new name
• appA needs to be rebuilts with these two packages.

I have planned to write you an email or maybe sync up with you somehow to describe what I did (maybe I did things wrong, that's possible) and what would be interesting for the conan project (or a companion project, that's also possible). My tools are not open source for the moment, but it is planned sometime this year :D

[lasote]

@gsemet thanks. Would be nice to know more about your solution and we will be glad to have a meeting if we can help. Without knowing many details about what you did, using package revisions and lock files you should be able to manage it. Have you taken a look at this docs? I think we might have more material to learn about it, I will search and update.

[gsemet]

Sure, I'll try to set up something for you. Yes I would like to use lock files ultimately, I use an intermediate file to store the component to test that looks like a mini-lock file (only direct level of dependency is overriden).

But I do not see how to use the lockfile directly, how can you patch the lock file of another project to say "if in your graph you have libA/1.0.0, take the libA/1.0.1-pre.1234@testing/my-dev-branch")

[KerstinKeller]

For me (I commented on the linked issue), this mainly stems from having transitive dependencies and later making releases.
We early adopted using version ranges only in recipes, as fixing versions tremendously increases integration effort.
In the end, noone but the final consumer will resolve exact versions.
In that case, I also would not call it "overriding" dependencies, but "resolving" dependencies.

From the conanfile of the end-consumer that fixes all versions of all dependencies to be used, lockfiles will be created and they are used to build packages in between. So at the moment, there is no need for us to have this override feature on command line any more.

[gsemet]

What I say is not in contradiction with your point. We do have version range in recipe (I would like to use lockfile only on the app).
My use case is just to test BEFORE merge. Once merged, everything happens according to what to do and probably everybody does.

[lasote]

Thanks! I'm getting a better picture now, still would like to receive more use cases from the users that upvoted the issue:

@yuri-kilochek @niosHD @driesdeschout @petermbauer @raffienficiaud @turkishviking @josh-stoddard-tanium @IceflowRE @ilya-v @lieser

[lieser]

We have two use cases, both mainly for our CI.

1. Build and test our own packages with different versions of the dependencies

Lets assume we have liba, for which both a stable version and an unstable version exist (e.g. liba/1.0.0 created from a tag, and liba/master which is created on each commit to master).

We also have a libb, which depends on liba. In the recipe of libb checked into git, we want to require on the stable version of liba, i.e. liba/1.0.0.

I don't thing its relevant here, but note that all our own packages are currently using full_package_mode().

Now in our CI, we want to build and test libb not just with the stable version of liba, but also against the unstable version of liba. And in the future, especially if we start using version ranges, we may additionally want to build and test against different stable versions of liba.

As these are our own packages, we were able to add some logic to the packages, which allows to override the references in the requires via environment variables.

So this is currently already possible, but we believe it would be good to have a common an officially supported approach here, and get the redundant logic out of the recipes.

2. Build and publish packages from conan-center with overridden requirements

We have our own internal CI using conan-package-tools for building some packages from conan-center.
There we want to pre-build the packages in the configurations used by consumers. And the consumers may want to override which version of a dependency is used, e.g. which version of openssl the libcurl package is using.

This is admittedly currently only a theoretically use case for us. We have not yet used Conan that long, and so far the default references inside the recipes worked out for us. But I imagine that will not always be the case.
And writing this, I also noticed that this would probably also only work if we changed the default package-id mode, which we currently do not do.

An alternative approach here would be to manually change the requires in the recipe from conan-center before building. And as long as we do not change the default package-id mode it is maybe also the better one.

---

Note that for us it would be important that the solution works well in combination with the conan-package-tools.

[KerstinKeller]

Ok, maybe I do revisit my comment from above. In general, this feature can be necessary for developers, that are debugging recipes.
My recipe has a

  self.requires("libA/[>=1.0.0]@user/stable")

Conan always takes the newest, and resolves it to 1.2.0, however, there are no prebuild binaries in the server (yet) for the profile I am using. However there are binaries for 1.1.0.
Either I now have to to modify the conanfile.py, or which would be preferable, I could do a conan install .. --requires=libA/1.1.0@user/stable which is just quick and convenient.
(Or conan lock create --requires=libA/1.1.0@user/stable -> if conan install supports it, lock create should do as well)

[petermbauer]

We have a similar use-case as already described by others:
"what-if" builds in CI to check if a new version of an upstream package would break something in downstream packages. Currently the workaround is based on environment variables overriding the requirements specified in the conanfile.py

[slietzau]

I agree that the mechanism would be a cool feature.

As a workaround it is possible to download the package in the desired version/revision and then trigger the conan install. If --update is not used, the version in the local cache will be used instead of one from the server. This allows to force a specific recipe revision to be used without specifying lockfiles.

[lasote]

Thanks all for answering.
We got the feedback and will try to come out with something experimental for the 1.39 because we are already late for 1.38.

[memsharded]

Proposing #9195 draft, to start experimenting with this.

[KerstinKeller]

Some additional side note:
The feature in the draft is now called -requires-override.
Maybe it makes sense to clarify this a bit more, and maybe to define what override means.

My understanding of override is to redefine. E.g. libB has a requirement of libA/1.0.0, and now I am forcing it to use a different version instead.

However, my usage (and from what I understand some of the other users have similar use cases), is just to provide a valid version when using version ranges, which will satisfy existing requirements, but not "override" anything.

So I don't know if it makes sense to distinguish between the two usecases or not.

[memsharded]

Hi @KerstinKeller

The proposal in #9195 only implements one of the existing overrides, which does not do version range checking upstream. It doesn't do it, because for regular requirements, it also does an override. If one package upstream depends on a fixed version dep/1.0@user/channel, it can be overriden from downstream to dep/2.0@user/channel, even if it is clearly outside of its defined "range" (no range). The same behavior applies for version ranges, the downstream always mandates on the upstream which version to use, even if outside the version ranges (it is the only way to always be able to solve conflict from downstream without editing recipes).

It would also totally make sense that this feature is used to try to validate if the next version outside of a range would work in the graph, without needing to modify the recipe.

If this is not the desired behavior of this feature, we would need to know, please @gsemet @wpalfi @petermbauer, @slietzau, @Minimonium, @mpdelbuono, @lieser, @vctgross, feedback very welcome.

[gsemet]

Hi. That would cover part of our needs (we also use this overrides for build_requirements). Hi does it behave for transitive dependencies or during export ?

For transitive dependencies, A depends on B and C and B depends on C, if C is overrrided, when B is rebuilt it needs to take the overridden C as well.

Also, the export triggers the “requirement” hook that I use to simulate this override, so we need to carry the wanted override as well.

basically after a Conan install with override, whatever happen, whatever range protection, we need to use the overridden lib in every case.

[Minimonium]

The current PR looks great for my use case. The by-passing behavior is what I need and it conforms to the current override behavior in other parts of Conan.

[lieser]

I would say that overriding without validating fits our use case better.
We would e.g. like to overwrite a version of dep/1.0.0@user/stable or dep/[^1.0.0]@user/stable with dep/1.1.0@user/stable or dep/master@user/testing.

[petermbauer]

I would say that overriding without validating fits our use case better.
We would e.g. like to overwrite a version of dep/1.0.0@user/stable or dep/[^1.0.0]@user/stable with dep/1.1.0@user/stable or dep/master@user/testing.

Same here, one often does not know in advance what may be overridden in the future and with what version/user/channel.

[memsharded]

Yes, the override is hard, as long as the package name is the same, the rest of the reference will be overriden.

For transitive dependencies, A depends on B and C and B depends on C, if C is overrrided, when B is rebuilt it needs to take the overridden C as well.

It also applies to transitive dependencies, yes, once a version is overriden it is overriden for the graph, not only for the declaring override consumer.

Also, the export triggers the “requirement” hook that I use to simulate this override, so we need to carry the wanted override as well.

Not sure what is the requirement hook (https://docs.conan.io/en/latest/reference/hooks.html#hooks-reference), so can't say about this.

basically after a Conan install with override, whatever happen, whatever range protection, we need to use the overridden lib in every case.

Yes, this is the behavior 👍

[KerstinKeller]

The proposed implementation is a good way to quickly force overriding certain dependencies.

Maybe in the future we can have a separate discussion on how users can influence the graph calculation in general, and what are best practices for modeling requirements (version ranges vs fixed versions) and how teams can handle / update large dependency graphs.

[memsharded]

I am going to merge #9195 for Conan 1.39.
Note that this include only regular requirements overrides, not build_requires overrides

The problem with build_requires is the resolution logic is very different. If we override the build_requires of the base recipe to which the command is referencing, that do not propagate upstream. If we override the build_requires of the profile, now we have 2 profiles, build and host. It would be necessary to specify which one. And it should be clear that this is not an override in the same sense, but a profile definition (it can apply and inject the build-require to all packages in the dependencies, not only override the version of the existing ones). Trying to implement an actual override of only existing versions, would also be a more complicated feature, it is not a UX one, where the logic is there and is only making it accessible on the command line, we would be talking about a complete new way of defining build_requires versions upstream.

As this issue will be automatically closed when the PR is merged, I suggest if you are interested in the build_requires case to open a new issue when we can discuss the specifics of this case. Thanks!

[gsemet]

Ok thanks. I have created #9302 for discussion about overriding "build_requires" dependencies.