Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate with Conan package manager #327

Closed
solvingj opened this Issue Jan 12, 2019 · 12 comments

Comments

Projects
None yet
3 participants
@solvingj
Copy link

solvingj commented Jan 12, 2019

Per @waruqi request, opening this ticket to track an effort to add support for Conan packages.

Ideally, each build system would not need to support each package manager, but this is not a solved problem yet. Until then, custom support must be added to find Conan packages installed on the local machine.

@waruqi waruqi added this to the v2.2.4 milestone Jan 12, 2019

@waruqi

This comment has been minimized.

Copy link
Member

waruqi commented Jan 12, 2019

Ok, I will support it.

@yssource

This comment has been minimized.

Copy link

yssource commented Jan 12, 2019

xmake is going to work as a generator in Conan, isn't it?

@waruqi

This comment has been minimized.

Copy link
Member

waruqi commented Jan 12, 2019

@solvingj I have supported to find conan packages using lib.detect.find_package. You can update xmake from dev branch and try it.

$ xmake l lib.detect.find_package bzip2

{
    links = 
    {
        bz2
    }
,   linkdirs = 
    {
        /Users/ruki/.conan/data/bzip2/1.0.6/conan/stable/package/534dcc368c999e07e81f146b3466b8f656ef1f55/lib
    }
,   includedirs = 
    {
        /Users/ruki/.conan/data/bzip2/1.0.6/conan/stable/package/534dcc368c999e07e81f146b3466b8f656ef1f55/include
    }
}

Find and use conan packages in project with xmake.lua

target("test")
    set_kind("binary")
    add_files("src/*.c")
    on_load(function (target)
        import("lib.detect.find_package")
        target:add(find_package("bzip2"))
    end)

And add_requires also support to find conan packages:

add_requires("bzip2")
target("test")
    set_kind("binary")
    add_files("src/*.c") 
    add_packages("bzip2")
@waruqi

This comment has been minimized.

Copy link
Member

waruqi commented Jan 12, 2019

@yssource Yes, we are planning to do it.

@solvingj

This comment has been minimized.

Copy link
Author

solvingj commented Jan 12, 2019

Thanks! Can you tell me how it finds packages? In theory, the user should have to include the generated conanbuildinfo.lua file, and then cmake could look for the targets with the Conan- prefix. Like, Conan-zlib, although we can decide how the names will be formatted when we create the generator.

@waruqi

This comment has been minimized.

Copy link
Member

waruqi commented Jan 12, 2019

@solvingj You can see 8062913

@solvingj

This comment has been minimized.

Copy link
Author

solvingj commented Jan 12, 2019

Ok, I have looked at the code and unfortunately, the way you have tried to solve the problem will not work. I will try my best to explain the problem, and how we'll need to proceed.

Build System Settings
As you know, every time a build system such as xmake runs a build (compiles and links), it does so with a specific set of "settings". For example:

  • MSVC 15 compiler, targeting x86 architecture, in Release mode, on Windows.
  • GCC 8 compiler, targeting x86_64, in Debug mode, on Linux
  • Etc...

Libraries generated with one set of these "settings" can generally only be linked into binaries being compiled with the same settings. Furthermore, compiler flags, preprocessor flags, and special library options might need to be different, depending on the current settings.

Also of note, developers often have to switch between different settings very frequently, going from Release to Debug, or from x86 to x64, etc. As a result, build systems generally create a unique folder structure for each unique combination of settings to hold the output files. Conan does something similar, but in a far more robust way. Also, it produces more than just folders with files, it produces critical variables for consuming build systems, such as defines, cflags, cppflags, libs, and more.

Conan Settings
Conan defines the following four default "settings", and each time a recipe is built and the package is stored, it's stored in a unique folder based on these settings:

  • os
  • arch
  • compiler
  • build_type

It does this each time Conan is run, by taking the values for all the settings (in the current execution) and hashing them to produce a unique_abi_hash.

Conan also lets packages define custom options for libraries, which will be factored into the unique_abi_hash, and thus be stored in separate folders. For example, many libraries add an option called shared, which lets users choose between the shared library binaries or the static library binaries.

Finally, as mentioned, each package might have advanced logic to choose variables that it will pass to consumers, including defines, cflags, cppflags, libs, and others. All of this is determined at execution time by a python function called package_info(). Here's the function defined by zlib package:

    def package_info(self):
        if self.options.minizip:
            self.cpp_info.libs.append('minizip')
            if self.options.shared:
                self.cpp_info.defines.append('MINIZIP_DLL')
        if self.settings.os == "Windows" and not tools.os_info.is_linux:
            if tools.os_info.is_windows:
                self.cpp_info.libs.append('zlib')
            else:
                self.cpp_info.libs.append('z')  # MSYS/Cygwin builds
        else:
            self.cpp_info.libs.append('z')

The Workflow of Integration
Due to these and other complexities, conan's generator system is the only way to consume conan packages from a build system like xmake. There can be some automation between xmake and Conan (link below), but in the end, xmake MUST work with the conan generator system by consuming it's files.

Manual Workflow
Here is an example of the process if conan is run manually, with no special "find_package" functionality added to xmake.
Note: this is intentionally verbose and explicit for demonstration:

  1. User wants to build gbox with xmake using settings: Linux/gcc8/x64/Release.
  2. User wants zlib static library from Conan.
  3. User creates conanfile.txt in the root of the gbox project, and lists dependencies:
[generators]
xmake
[requires]
zlib/1.2.11@conan/stable
  1. User creates a unique folder in some build directory for gbox, for the desired settings. Example:
    build/linux_gcc8_x64_release

  2. User runs the following command:
    conan install . <username>/testing -s os=Linux -s compiler=gcc -s compiler.version=8 -s arch=x86_64 -s build_type=Release -o zlib:shared=False --install-folder="build/linux_gcc8_x64_release"

  3. This produces the following file:
    build/linux_gcc8_x64_release/conanbuildinfo.xmake.lua
    Note: remember, the paths and variables are specific to the settings used (Linux/gcc8/x64/Release), and only containing the specific dependencies of gbox, as opposed to some global cache of all packages from the whole system.

  4. The user adds a line to their main xmake build file, something like:
    if is_plat("linux") and is_arch("x86_64") and is_comp("gcc") and is_comp_ver("8") and is_mode("release") then
    add_include("build/$(platform)_$(compiler)$(compiler_version)_$(arch)_$(configuration)")

Now when the user runs the xmake build, it will have access to all the appropriate variables and values for zlib.

Improving the workflow

Obviously, the above example is extremely naive and unnecessarily verbose. I used it because it shows how all the variables are relevant in both xmake and conan, and it demonstrates the necessity of making sure both tools are using the same settings information all the time.

To improve the flow, there are several things that can be done.

Here's an integration for Cmake, implemented as a CMake module. With this approach, CMake did not have to be modified at all. Users simply have to include this module in every project which wants to use Conan. It has some disadvantages, but it's an option for xmake. At the very least, it will be helpful for you to look at to understand how to interact with Conan programatically.
https://github.com/conan-io/cmake-conan/blob/develop/conan.cmake#L29

However, since you're already writing custom code for conan directly into the xmake platform, here is another option, which is much more streamlined. You could add an environment variable or command-line-parameter to xmake. Something like:
XMAKE_CONAN_ENABLE/--conan-enabled

If the user sets or passes this option when running xmake then you can perform the following automatic steps:

  • Search for conanfile.txt or conanfile.py in the project root (for starters)
  • If found, execute conan install . , passing xmake settings to Conan (refer to conan.cmake)
  • For --install-folder, pass xmakes current temporary build directory
  • Automatically add_include() the generated conanbuildinfo.xmake.lua to the project
  • For the find_library behavior, if the user wants zlib, search CONAN_ZLIB as an alias for zlib.

** Summary **
I think this last approach is the best. It uses Conan the way it's intended, and requires minimal code within xmake. It ensures xmake is always using the correct, and current Conan paths and variables for the current xmake execution, and takes advantage of xmake's unique directories to store generated files. This way, users don't have to manage unique directories just to hold generated files themselves (like in the manual example i gave).

I will start work on the xmake generator now. It should not take long.

@waruqi

This comment has been minimized.

Copy link
Member

waruqi commented Jan 13, 2019

Ok, I will improve it.

@solvingj

This comment has been minimized.

Copy link
Author

solvingj commented Jan 13, 2019

Great, thanks! See #331 for the working draft of the xmake_generator.

@waruqi waruqi changed the title Add custom support for finding packages from Conan package manager Integrate with Conan package manager Jan 14, 2019

@waruqi waruqi pinned this issue Jan 14, 2019

@waruqi

This comment has been minimized.

Copy link
Member

waruqi commented Jan 29, 2019

I improved find_package and re-integrated conan, you can see (in dev branch) https://github.com/tboox/xmake/tree/dev/xmake/modules/package/manager/conan

add_requires("CONAN::zlib/1.2.11@conan/stable", {alias = "zlib", debug = true, 
    configs = {build_requires = "xmake_generator/0.1.0@bincrafters/testing", build = "all"}})

add_requires("CONAN::OpenSSL/1.0.2n@conan/stable", {alias = "openssl", 
    configs = {build_requires = "xmake_generator/0.1.0@bincrafters/testing",
    options = "OpenSSL:shared=True", build = "all"}})

target("test")
    set_kind("binary")
    add_files("src/*.c") 
    add_packages("openssl", "zlib")

Build project

ruki:test_package ruki$ xmake
checking for the architecture ... x86_64
checking for the Xcode directory ... /Applications/Xcode.app
checking for the SDK version of Xcode ... 10.14
note: try installing these packages (pass -y to skip confirm)?
  -> CONAN::zlib/1.2.11@conan/stable  (debug)
  -> CONAN::OpenSSL/1.0.2n@conan/stable  
please input: y (y/n)

  => installing CONAN::zlib/1.2.11@conan/stable .. ok
  => installing CONAN::OpenSSL/1.0.2n@conan/stable .. ok

[  0%]: ccache compiling.release src/main.c
[100%]: linking.release test

If you want to known more, you can see #339

@solvingj

This comment has been minimized.

Copy link
Author

solvingj commented Jan 30, 2019

this is looking very good, amazing progress as usual. it seems you have learned a great deal about Conan, and now have plans on how to improve your support for it in the future. I think continued improvement on this integration will be sufficiently tracked in #339 along with the other package managers in a general way. Overall, there is a mostly-working integration now, so I will close this ticket.

@solvingj solvingj closed this Jan 30, 2019

@waruqi waruqi unpinned this issue Jan 30, 2019

@waruqi

This comment has been minimized.

Copy link
Member

waruqi commented Mar 29, 2019

A simple example

add_requires("CONAN::zlib/1.2.11@conan/stable", {alias = "zlib", debug = true}})
add_requires("CONAN::OpenSSL/1.0.2n@conan/stable", {alias = "openssl", 
    configs = {options = "OpenSSL:shared=True"}})

target("test")
    set_kind("binary")
    add_files("src/*.c") 
    add_packages("openssl", "zlib")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.