[question] How to handle products that uses same dependency tree with different versions

[Schmeichael]

What is your question?

Hello!

I need your assistance please.
I will describe my question with a toy example of the issue.

let's say, my team develops product named prod1.
to achieve that it uses 2 dependencies: A and B.
B also depends on A.
so, let's say prod/1.0 depends on A/[>=1.0 <2] and B/[>=1.0 <2] and B/1.0 depends on A/[>=1.0 <2]

now, let's say we develop new product named prod2.
we created several breaking changes on A and therefore we bump major version and create A/2.0 - we will now only deliver bug fixes for A/1.*.*
B had to fix some A's API in order to build and therefore we bump minor version and create B/1.1.
so, the requirements for prod2/1.0 will be A/[>=2.0 <3] and 'B/[>=1.1 <2]'

now, prod2 builds OK. with the latest A and B to support it needs.
but --- prod1 can't be built with 'B/[>=1.0 <2]', because Conan will resolve 'B/[>=1.0 <2]' to B/1.1 and will create conflict with A

• how should this use case be handled?
• should we now lower the upper boundary of B, in prod1?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @Schmeichael

Thanks for your question.

I think a possible approach should be the one reflected in https://docs.conan.io/2/ci_tutorial/tutorial.html, in the "products" pipeline, I don't know if you already had a look.

The idea is that you have 2 products, prod1 and prod2. When you are doing a change in the graph, that change needs to be validated against all products you want to keep maintained. In this case it seems both prod1 and prod2.

So you initially have this graph for prod1:

prod1 ------------ A/[>=1.0 <2] ----------------> A/1.1
    \----- B/[>=1.0 <2  -> B/1.0--(A/[>=1.0 <2] ---/

The new prod2 comes out, lets say it starts without dependencies.:

prod2

Then A/2.0 is done and comes out but it doesn't break anything yet, because it is excluded by the version ranges, all ok so far.

The problem happens if we do a B/1.1 depends on A/2.0, and accept it as good without validating in our products pipeline against all of our products.

If we do that, we will get an error, probably a version conflict, but it could also be compilation errors due t the new API.

Our "products pipeline" tells that the B/1.1 version would be breaking our products, because of the dependency to A/2.0.
In practice this means that the C/C++ compilation and ABI model can break consumers even for API compatible changes (prod1 is not affected by the API changes in B/1.1, which remains stable, but will be broken due to the transitive A/2.0.

This breakage could be limited or contained by different approaches, for example if the relationship B->A is made fully private (no API neither ABI visibility, then using the visible=False trait, then B is a shared library and A a static library), it could be unaffected, but in the majority of cases this is not desired at all. It is possible that in some cases even the headers of A need to be propagated to the prod1 via transitive_headers=True trait, because they are included in the public headers of B.

So the general solution should be that changes in packages, such as B/1.1 needs to be validated against all of our products. If one of our products rejects it because it is breaking, it means that B/1.1 will not move forward and won't be integrated, and should probably also get a major version such as B/2.0, so it can be moved downstream in a more controlled way.

There could be a different approach that could make more sense if prod1 and prod2 are more decoupled, like developed by different teams. It could be possible to split the server repositories, create one repository for the prod2 team, so they resolve dependencies against it, and put A/2.0 and B/1.1 there, so the developers at prod1 do not use that repo and are unaffected by the changes. Then, if at some point we want to extend those new versions to prod1, we could start testing those packages against prod1, and doing the necessary changes as outlined above until it is possible to integrate them.

Please let us know if this helps.

[Schmeichael]

thanks @memsharded for the response.

Private relationship Between A -> B

as you mentioned, it's not desired and feasible in our use case.

Major version bump

I don't like it, especially when it's possible scenario that B didn't had to change any of its sources (for example when it doesn't hit any A API change). that means that B/1.0 can get built against A/1... and A/2.... and doesn't require any version bump, as the package should reflect its source code. we thought maybe we could add to our versioning system a digit representing requirements change, so it would be: REQUIREMENTS_ID.MAJOR.MINOR.PATCH but I don't like it either because it seems this mechanism is already embedded into conan package id and it breaks SEMVER. what do you think? to be honest, we kinda hoped that the dependencies are going to be handled like compiler / os / arch settings. means that we can create same revision of a package using different requirements versions.

Splitting to different server repo's

that's reasonable solution. though, A package (and some more packages in our build tree) getting major bump every quarter, so I don't think it scales (or am I wrong?). currently we use 2 server repos, 1 for releases and 1 for testing and integrations.

I have also another solution in mind:

Use user attribute

I know using the user attribute is discouraged, but for our use-case it seems kinda proper solution.
what are the pros and cons comparing to the different repo's solution?

[Schmeichael]

Any thoughts on that ?

[memsharded]

Sorry for the delay

I don't like it, especially when it's possible scenario that B didn't had to change any of its sources (for example when it doesn't hit any A API change). that means that B/1.0 can get built against A/1... and A/2.... and doesn't require any version bump, as the package should reflect its source code. we thought maybe we could add to our versioning system a digit representing requirements change, so it would be: REQUIREMENTS_ID.MAJOR.MINOR.PATCH but I don't like it either because it seems this mechanism is already embedded into conan package id and it breaks SEMVER. what do you think? to be honest, we kinda hoped that the dependencies are going to be handled like compiler / os / arch settings. means that we can create same revision of a package using different requirements versions.

That is fine, I understand that you don't want to bump B/1.0 to a major version.

It is correct, Conan does support B/1.0 or B/1.1 building both against A/1.0 and A/2.0, and it will correctly manage the different binaries without problems.

But this is not what the original report said:

B had to fix some A's API in order to build and therefore we bump minor version and create B/1.1.
so, the requirements for prod2/1.0 will be A/[>=2.0 <3] and 'B/[>=1.1 <2]'

If B/1.1 does a A/[>=2.0 that is exclusive of A/1.0, and then the version conflict.
The only solution to not break prod1 is to also update the requires to A in prod1 too, otherwise, there is an unsolvable version conflict.
If as you said. B/1.0 adds range for A/[>=1.0 <3] then, it is possible under some circumstances, that the graph for prod1 will still be correctly resolved to A/1.0.

that's reasonable solution. though, A package (and some more packages in our build tree) getting major bump every quarter, so I don't think it scales (or am I wrong?). currently we use 2 server repos, 1 for releases and 1 for testing and integrations.

That is not a problem. There are many organizations creating repos even for every Pull Request. Creating a new repo per quarterly release shouldn't be a big issue.

I know using the user attribute is discouraged, but for our use-case it seems kinda proper solution.
what are the pros and cons comparing to the different repo's solution?

Yes, that would be one of the discouraged cases. Conflicts happen the same, because the user is just a qualifier, but no 2 packages with the same package name can be in the same dependency graph. pkg/1.0@user and pkg/2.0@other will still conflict if in the same dependency graph.

[memsharded]

Any feedback or question about the last comment?
Thanks for your feedback!

[AbrilRBS]

Closing now as we didn't hear back from you, please do feel free to reopen/create a new issue if you have any other questions, thanks!