Package dependencies are not transitive

[SirLynix]

Is your feature request related to a problem? Please describe.

Hello,

As described here, the way package dependencies are handled can cause subtle bugs.

For example

add_requires("libsdl 2.28", "libsdl_ttf")

this will install two different versions of libsdl, because libsdl_ttf doesn't specify a version:

checking for Microsoft Visual Studio (x64) version ... 2022
note: install or modify (m) these packages (pass -y to skip confirm)?
in xmake-repo:
  -> libsdl 2.28.5 [runtimes:"MT"]
  -> libsdl#1 2.30.8 [runtimes:"MT", from:libsdl_ttf]
  -> libsdl_ttf 2.22.0 [runtimes:"MT"]
please input: y (y/n/m)

this may duplicate symbols, fail compilation or even worse, introduce subtle bugs that will just prevent the application to work without failing compilation.

An example would be something that happened countless times to my students:

package("student_engine", function ()
    add_deps("libsdl")
end)

add_requires("student_engine")
add_requires("libsdl", { configs = { shared = true }})

here we have two versions of libsdl, one static linked to student_engine and one shared linked to the main app, which was causing bugs because libsdl uses global variables internally, they were initializing it in the engine but SDL calls were failing in the app because it wasn't the same libsdl instance.

Another example would be openssl, a common package dependency, which restricts you to use multiple openssl versions across the project, if you use package locking or simply if you set the package version you can get multiple openssl versions.

Describe the solution you'd like

I think xmake should try to merge configs whenever possible, for example:

package("student_engine", function ()
    add_deps("libsdl")
end)

add_requires("student_engine")
add_requires("libsdl", { configs = { shared = true }})

here, student_engine.libsdl doesn't care about being shared or not, so we could merge both configs.

however:

package("student_engine", function ()
    add_deps("libsdl", { configs = { shared = false })
end)

add_requires("student_engine")
add_requires("libsdl", { configs = { shared = true }})

there should be two instances of libsdl here (or an error).

A quick way to do this would be to make add_requires("foo x.y", { configs = { ... }) implicitly add an add_requireconfs("**.foo", { version = "x.y", configs = { ... })` call.

Since it's a breaking change, this could be a policy, but it seems like a good default for xmake 3.0.

Describe alternatives you've considered

Using add_requireconfs by yourself is a way to fix it, however this can lead to more complex xmake.lua and you also need to understand how packages depend on each other, which is not simple.

Simpler xmake should be less error-prone in my opinion.

Additional context

Note that vcpkg does something like this at least for versions: https://learn.microsoft.com/en-us/vcpkg/users/versioning.concepts

[xq114]

add_requires("libsdl", { configs = { shared = true }, public = true}) Add an argument to indicate passing it to other packages is also ok

[waruqi]

add_requires("libsdl", { configs = { shared = true }, public = true}) Add an argument to indicate passing it to other packages is also ok

if public libsdl will modify A and B's libsdl dep configs, foo and bar will both use shared libsdl.

But some users maybe only want to modify foo -> A -> libsdl to shared, bar/B still use static libsdl.

Therefore, this global modification way has too much impact and is not very flexible. it will broke something.

I think the top-level add_requires should be completely independent and independent of each other. Then we can use add_requireconfs to control their dependency configuration more flexibly and finely.

add_requires("A") -- dep: libsdl
add_requires("B") -- dep: libsdl
add_requires("libsdl", { configs = { shared = true }, public = true})

target("foo")
    add_packages("A")

target("bar")
    add_packages("B")

[SirLynix]

I think that xmake default configuration should be the one that lead to minimal library duplication, as it can bloat the app or cause compilation or even runtime bugs (as with libsdl and openssl).

the issue with add_requireconfs is that it requires advance knowledge of how xmake works and to know which libraries depend on each other.

you may want to use openssl 3.2 without being aware that some package you also use will require openssl, leading to duplication (which can even go unnoticed as maybe xmake will find openssl 3.4 on the system).

Therefore, this global modification way has too much impact and is not very flexible. it will broke something.

Yes it may break existing project, which is why I think we should introduce a policy for that and furthermore that this policy should be the default behavior on xmake 3.0

[waruqi]

the issue with add_requireconfs is that it requires advance knowledge of how xmake works and to know which libraries depend on each other.

just add_requireconfs("**.libsdl", {configs = {shared = true}}), we need not to know more.

Yes it may break existing project, which is why I think we should introduce a policy for that and furthermore that this policy should be the default behavior on xmake 3.0

I don't think it's a good idea to make this the default behavior for 3.0. It will lead to additional limitations in some package dependency configurations. As I said above.

[SirLynix]

just add_requireconfs("**.libsdl", {configs = {shared = true}}), we need not to know more.

After introducing xmake to my students for four years I can guarantee your that even just configuring libraries in add_requires needs some preparation, add_requireconfs is advanced xmake usage for them.

I don't think it's a good idea to make this the default behavior for 3.0. It will lead to additional limitations in some package dependency configurations. As I said above.

I don't understand why there would be limitations, since add_requireconfs can override each other you can always undo the default behavior using add_requireconfs.

The point of this feature request is to make the default behavior of xmake less error-prone, having add_requires("foo", configs) automatically do an add_requireconfs("**.foo", configs) would be a simple way to fix some of these issues.

[xq114]

add_requires("libsdl", { configs = { shared = true }, public = true}) Add an argument to indicate passing it to other packages is also ok

if public libsdl will modify A and B's libsdl dep configs, foo and bar will both use shared libsdl.

But some users maybe only want to modify foo -> A -> libsdl to shared, bar/B still use static libsdl.

Why not write like this?

add_requires("libsdl", { configs = { shared = true }, public = true})
add_requires("A") -- dep: libsdl
add_requires("B") -- dep: libsdl
add_requireconfs("B.libsdl", { configs = { shared = false }})

target("foo")
    add_packages("A")

target("bar")
    add_packages("B")

Since B.libsdl is more specific then **.libsdl, the option in B.libsdl should override that in global **.libsdl, just like C++ concept or CSS selector.

[waruqi]

When foo/bar only depend on packages A and B, even for beginners, their initial configuration will only think of the following configuration.

add_requires("A") -- dep: libsdl
add_requires("B") -- dep: libsdl

target("foo")
    add_packages("A")

target("bar")
    add_packages("B")

Therefore, they will still encounter libsdl dependency conflicts, or want to modify the configuration of libsdl to use dynamic libraries.

Then they spend the same amount of time figuring out whether to use add_requires or add_requireconfs , or add_requires + public to solve it.

Maybe using add_requires will be slightly faster, but it will bring some global side effects, causing other package dependencies to be modified as well. Users still need to spend more time studying how to use add_requireconfs to modify them again.

[waruqi]

add_requires("libsdl", { configs = { shared = true }, public = true}) Add an argument to indicate passing it to other packages is also ok

if public libsdl will modify A and B's libsdl dep configs, foo and bar will both use shared libsdl.
But some users maybe only want to modify foo -> A -> libsdl to shared, bar/B still use static libsdl.

Why not write like this?

add_requires("libsdl", { configs = { shared = true }, public = true})
add_requires("A") -- dep: libsdl
add_requires("B") -- dep: libsdl
add_requireconfs("B.libsdl", { configs = { shared = false }})

target("foo")
    add_packages("A")

target("bar")
    add_packages("B")

Since is more specific then , the option in should override that in global , just like C++ concept or CSS selector.B.libsdl``**.libsdl``B.libsdl``**.libsdl

For the user, the time to read the documentation to solve the problem is the same, but the following configuration is simpler.

add_requires("A") -- dep: libsdl
add_requires("B") -- dep: libsdl
add_requireconfs("A.libsdl", { configs = { shared = true }})

[waruqi]

I think a better way is to automatically solve most dependency conflicts without adding any additional configuration. For example, in the same dependency chain, if there are multiple packages with the same name but different configurations, xmake can automatically select the package configuration with the best compatibility.

Automatically select compatible version

This allows us to automatically resolve such issues without requiring the user to add additional add_requires configuration. https://github.com/xmake-io/xmake-repo/pull/5465/files

Maybe we need to improve package configuration to provide more information about compatibility between versions. But I haven't figured out how to do that yet.

Maybe we need to add a new API for packages to describe compatibility information of dependent packages, including (versions, feature configurations, runtimes, etc.)

e.g. A -> libsdl 2.0 -> C -> libsdl 2.8

result: A -> C -> libsdl 2.8

like this

package(A)
    add_deps("libsdl", {compatibility = {version = ">=2.0 <=2.8"}})

If there is no compatibility configuration, all versions are compatible by default.

We can also provide more flexible compatibility strategies based on the current package version.

package(A)
    add_deps("libsdl", {compatibility = function (package)
       if package:version_str() == "1.0" then
            return {version = ">=2.0 <=2.8"}
       end
    end})

Of course, this is not the final API, but just some of my current preliminary design ideas.

Automatically select package configuration compatibility

e.g. A -> boost[filesystem] -> C -> boost[filesystem,asio]

result: A -> C -> boost[filesystem,asio]

Dependency conflicts that cannot be automatically fixed

I hope that automatic repair can solve 80% of dependency problems. In fact, most dependency conflicts are caused by version conflicts.

If the remaining 20% ​​of cases cannot be automatically fixed, we can improve the conflict error prompts and guide users to use add_requireconfs to manually fix them.

[SirLynix]

This seems great! Would the new compatibility syntax be required or would it be automatically taken into account when a package has such deps: add_deps("libsdl >=2.28") or would add_deps("libsdl", {compatibility = {version = ">=2.28"}} be required?

[waruqi]

This seems great! Would the new compatibility syntax be required or would it be automatically taken into account when a package has such deps: or would be required?add_deps("libsdl >=2.28")``add_deps("libsdl", {compatibility = {version = ">=2.28"}}

If version compatibility automatic parsing is supported, we can use it directly add_deps("libsdl >=2.28 <=3.0")

[waruqi]

But I need to point out that the automatic repair of dependency compatibility can fix the following conflict errors (with version, configs ...)

"package(%s): conflict dependencies with package in ...

But it cannot solve the case mentioned at the beginning of this issue. Because it is not an issue, it is just a usage problem.

As described #5527 (comment), the way package dependencies are handled can cause subtle bugs.
For example
add_requires("libsdl 2.28", "libsdl_ttf")
this will install two different versions of libsdl, because libsdl_ttf doesn't specify a version:
checking for Microsoft Visual Studio (x64) version ... 2022
note: install or modify (m) these packages (pass -y to skip confirm)?
in xmake-repo:
-> libsdl 2.28.5 [runtimes:"MT"]
-> libsdl#1 2.30.8 [runtimes:"MT", from:libsdl_ttf]
-> libsdl_ttf 2.22.0 [runtimes:"MT"]
please input: y (y/n/m)
this may duplicate symbols, fail compilation or even worse, introduce subtle bugs that will just prevent the application to work without failing compilation.

libsdl and libsdl_ttf are both configured at the top level. They are completely independent packages. They will not encounter symbol conflicts when applied to different targets. This is a common usage. Different targets use different versions of sdl

add_requires("libsdl 2.28", "libsdl_ttf")

target("A")
     add_packages("libsdl")

target("B")
    add_packages("libsdl_ttf")

If we only apply them to one target, we should configure it like this instead of adding two add_requires at the same time

add_requires("libsdl_ttf")

target("A")
     add_packages("libsdl_ttf") -- it will also link libsdl

Therefore, I can try to implement automatic repair of package dependency conflicts in 3.0, but I cannot guarantee that the case mentioned in the current issue will be solved because it is not a bug, but just incorrect usage.