Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiple compilers, ConanMultiPackager and CMake troubles #124

Closed
theodelrieu opened this issue Dec 5, 2017 · 25 comments
Closed

Multiple compilers, ConanMultiPackager and CMake troubles #124

theodelrieu opened this issue Dec 5, 2017 · 25 comments
Milestone

Comments

@theodelrieu
Copy link

Hi!

I wish to package multiple packages for Linux x64, compiler with GCC/Clang.

To do so, I have two terminals. One has env vars CONAN_GCC_VERSIONS & cie, the other has the Clang variables.

The problem is that the chosen compiler (clang) is not detected by CMake when running test_package.
Here is the output I get:

-- The C compiler identification is GNU 7.2.0
-- The CXX compiler identification is GNU 7.2.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Conan: called by CMake conan helper
CMake Error at build/9fad4defe7e3af96f74f9172f136f8e10e7bf90a/conanbuildinfo.cmake:381 (message):
Incorrect 'clang', is not the one detected by CMake: 'GNU'

When creating a single package with conan create theo/stable -pr clang50, everything works fine:

-- The C compiler identification is Clang 5.0.0
-- The CXX compiler identification is Clang 5.0.0
-- Check for working C compiler: /usr/bin/clang
-- Check for working C compiler: /usr/bin/clang -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/clang++
-- Check for working CXX compiler: /usr/bin/clang++ -- works

@uilianries
Copy link
Member

Yes, you found a bug on conanclang50.

I'll provide a fix ASAP.

@theodelrieu
Copy link
Author

Thanks!

@uilianries
Copy link
Member

Hi again!

I made a PR including a new fix: conan-io/conan-docker-tools#28

Cheers!

@theodelrieu
Copy link
Author

Hi,

I'm not sure if that will fix it in my case, since I'm not using Docker at all, but simply running a build.py script locally.

@uilianries
Copy link
Member

uilianries commented Dec 6, 2017

So, do you have both compilers on same environment?
Are you using test_package command or ConanMultiPackager implementation in build.py?

I asking because both CONAN_GCC_VERSIONS and CONAN_CLANG_VERSIONS are only consumed by ConanMultiPackager.
Indeed, test_package will use your default profile if you don't declare an explicit profile.

On the other hand, ConanMultiPackager will create temporary profiles based on each job, including for both compilers.
I saw some similar situation, when I don't export CC=clang and CXX=clang++. When this occurs, both CMake and autotools prefer to use gcc instead of clang, even my profile is for clang.

BTW, test_package is deprecated, it was replaced by create.

Regards.

@theodelrieu
Copy link
Author

Yes I have both compilers on my environment, I am using the following script:

from conan.packager import ConanMultiPackager
import os, copy


# Do not forget to add correct env vars: https://github.com/conan-io/conan-package-tools

def _relwithdebinfo(build):
    b = copy.deepcopy(build)
    b.settings["build_type"] = "RelWithDebInfo"
    return b


def relwithdebinfo_builds(vs_builds):
    release_builds = list(filter(lambda build : build.settings["build_type"] == "Release", vs_builds))
    return map(_relwithdebinfo, release_builds)


if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds(shared_option_name="zlib:shared", pure_c=True)
    if os.name == "nt":
        builder.builds.extend(relwithdebinfo_builds(builder.builds))
    builder.run()

I tried to export CC, CXX, CMAKE_C_COMPILER, CMAKE_CXX_COMPILER with no success.

@uilianries
Copy link
Member

I found a very similar issue: conan-io/conan#1211

So, I customized build.py:

from conan.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager()
    builder.add_common_builds(shared_option_name="zlib:shared", pure_c=True)
    for settings, options, env_vars, build_requires in builder.builds:
        if settings["compiler"] == "clang":
            env_vars["CC"] = "/usr/bin/clang"
            env_vars["CXX"] = "/usr/bin/clang++"
    builder.run()

This works fine to build zlib running both clang and gcc on same environment.
Also, I don't have CC, CXX, CMAKE_C_COMPILER, CMAKE_CXX_COMPILER on my environment and I don't have /usr/bin/cc and /usr/bin/c++ on my PATH.

@theodelrieu
Copy link
Author

It's a bit of a hack but it works :p.

I've printed the temporary profile that gets generated:

DEBUG:  
include(gcc72)

[settings]
arch=x86_64
build_type=Debug
compiler=clang
compiler.version=5.0
[options]
[env]

[build_requires]

gcc72 is indeed my default profile. To solve this problem, I think adding a profile argument to ConanMultiPackager (defaulting to default) would be a good solution. I could then use argparse in the build.py to add a --profile, -pr option when invoking the script.

What do you think about it?

@uilianries
Copy link
Member

ConanMultiPackager provides a profile based on your settings, so can add what we need without force a profile argument. You could create your own builds:

from conan.packager import ConanMultiPackager

if __name__ == "__main__":
    builder = ConanMultiPackager(username="theodelrieu", reference="zlib/1.2.11")
    builder.add(settings={"arch": "x86_64", "build_type": "Release", "compiler": "gcc", "compiler.version": "7.2", "compiler.libcxx": "libstdc++"}, options={"zlib:shared": False}, env_vars={}, build_requires={})
    builder.add(settings={"arch": "x86_64", "build_type": "Release", "compiler": "clang", "compiler.version": "5.0", "compiler.libcxx": "libc++"}, options={"zlib:shared": False}, env_vars={"CC": "/usr/bin/clang", "CXX": "/usr/bin/clang++"}, build_requires={})
    # Add more builds ...
    builder.run()

@theodelrieu
Copy link
Author

I could do that, but I find more convenient to filter out the builds added by add_common_builds.

The advantage is that there is no need to add lots of if...else in the same script, nor forcing people to have both compilers on Linux for example.

Another solution would be to have different build scripts, build_clang.py, build_gcc.py etc...

@uilianries
Copy link
Member

I agree with you. I prefer to use add_common_builds and filter what I need.

@theodelrieu
Copy link
Author

That's why I'd like to tell ConanMultiPackager which profile it should inherit from, instead of defaulting to default.

I wonder what @lasote thinks about that :)

@lasote
Copy link
Contributor

lasote commented Dec 7, 2017

The @uilianries fix to iterate the builds and alter the env sounds good to me, not very hacky.
I'm going to try the approach of passing a profile and will report again.

@theodelrieu
Copy link
Author

Maybe adding an environment variable CONAN_DEFAULT_PROFILE would do the trick, adding a profile argument to the ConanMultiPackager requires to parse the command line, which is quite clunky in fact...

@lasote
Copy link
Contributor

lasote commented Dec 7, 2017

That would be a conan feature, more than a conan_package_tools, but for docker would require to pass it to the container.
I will upload a new version in a minute with the profile support as a parameter, then you can retrieve the environment variable value in the build.py script.

@lasote
Copy link
Contributor

lasote commented Dec 7, 2017

https://github.com/conan-io/conan-package-tools#specifying-a-different-base-profile

Upgrade and try!

@theodelrieu
Copy link
Author

Works perfectly! Thanks a lot.

@bilke
Copy link
Contributor

bilke commented Oct 17, 2018

@lasote I think that specifying a different base profile does not solve the problem which was originally stated. The test package step should set the environment variables CC and CXX in respect to the current profile from ConanMultiPackager. Otherwise CMake does not use the correct compiler and fails because of the Conan check (e.g. Incorrect 'apple-clang' version 'compiler.version=10.0' is not the one detected by CMake: 'Clang=9.1').

@lasote
Copy link
Contributor

lasote commented Oct 18, 2018

Ok, I will take a look again. Reopening the issue.

@lasote lasote reopened this Oct 18, 2018
@lasote lasote added this to the 0.20.0 milestone Oct 18, 2018
@lasote
Copy link
Contributor

lasote commented Oct 18, 2018

I don't' get it, sorry. Could you provide an example to reproduce the error? What do you have in your build.py? are you passing a default profile? With which contents?

@bilke
Copy link
Contributor

bilke commented Oct 18, 2018

I build Qt on macOS on Azure Pipelines. The vm image has several Xcodes installed. It seems the default one is Xcode 9 where the build runs fine. But when building for Xcode 10 the build of Qt is fine with this profile:

+-----------------------+
| Profile               |
|-----------------------|
| include(default)      |
|                       |
| [settings]            |
| arch=x86_64           |
| build_type=Release    |
| compiler=apple-clang  |
| compiler.version=10.0 |
| [options]             |
| Qt:shared=True        |
| qt:opengl=dynamic     |
| qt:qtxmlpatterns=True |
| [env]                 |
|                       |
| [build_requires]      |
+-----------------------+

But the test package-step fails:

Qt/5.11.2@bilke/stable (test package): Generator cmake created conanbuildinfo.cmake
Qt/5.11.2@bilke/stable (test package): Generator qt created qt.conf
Qt/5.11.2@bilke/stable (test package): Generator txt created conanbuildinfo.txt
Qt/5.11.2@bilke/stable (test package): Generated conaninfo.txt
Qt/5.11.2@bilke/stable (test package): Running build()
Qt/5.11.2@bilke/stable (test package): Building with qmake
Info: creating stash file /Users/vsts/agent/2.140.2/work/1/s/test_package/build/0e376c74ccf84edcb10e352d48e794bcc8ec68e4/qmake_folder/.qmake.stash
/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11  -arch x86_64 -isysroot /Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB -I../../../../test_package -I. -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include/QtGui -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include/QtCore -I. -I/Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/OpenGL.framework/Headers -I/Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/AGL.framework/Headers -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/mkspecs/macx-clang -o test_package.o ../../../test_package.cpp
/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -pipe -stdlib=libc++ -O2 -std=gnu++11  -arch x86_64 -isysroot /Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.11 -Wall -W -dM -E -o moc_predefs.h /Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/mkspecs/features/data/dummy.cpp
/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/bin/moc -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB --include /Users/vsts/agent/2.140.2/work/1/s/test_package/build/0e376c74ccf84edcb10e352d48e794bcc8ec68e4/qmake_folder/moc_predefs.h -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/mkspecs/macx-clang -I/Users/vsts/agent/2.140.2/work/1/s/test_package -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include/QtGui -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include/QtCore -I. -I/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1 -I/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.1.0/include -I/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include -I/Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include ../../../greeter.h -o moc_greeter.cpp
/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -c -pipe -stdlib=libc++ -O2 -std=gnu++11  -arch x86_64 -isysroot /Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.11 -Wall -W -fPIC -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB -I../../../../test_package -I. -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include/QtGui -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/include/QtCore -I. -I/Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/OpenGL.framework/Headers -I/Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/AGL.framework/Headers -I/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/mkspecs/macx-clang -o moc_greeter.o moc_greeter.cpp
/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ -stdlib=libc++ -headerpad_max_install_names  -arch x86_64 -Wl,-syslibroot,/Applications/Xcode_9.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -mmacosx-version-min=10.11 -Wl,-rpath,@executable_path/Frameworks -Wl,-rpath,/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/lib -o test_package.app/Contents/MacOS/test_package test_package.o moc_greeter.o   -L/Users/vsts/.conan/data/Qt/5.11.2/bilke/stable/package/7e524702348e5d10c19bbefa2feff2704284c8e6/lib -lQt5Gui -framework DiskArbitration -framework IOKit -lQt5Core -framework OpenGL -framework AGL 
Qt/5.11.2@bilke/stable (test package): Building with CMake
-- The C compiler identification is AppleClang 9.1.0.9020039
-- The CXX compiler identification is AppleClang 9.1.0.9020039
-- Check for working C compiler: /Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Conan: called by CMake conan helper
CMake Error at build/0e376c74ccf84edcb10e352d48e794bcc8ec68e4/conanbuildinfo.cmake:352 (message):
  Incorrect 'apple-clang' version 'compiler.version=10.0' is not the one
  detected by CMake: 'Clang=9.1'
Call Stack (most recent call first):
  build/0e376c74ccf84edcb10e352d48e794bcc8ec68e4/conanbuildinfo.cmake:409 (conan_error_compiler_version)
  build/0e376c74ccf84edcb10e352d48e794bcc8ec68e4/conanbuildinfo.cmake:480 (check_compiler_version)
  build/0e376c74ccf84edcb10e352d48e794bcc8ec68e4/conanbuildinfo.cmake:117 (conan_check_compiler)
  CMakeLists.txt:7 (conan_basic_setup)


-- Configuring incomplete, errors occurred!

See the full logs here and the recipe here.

In the profile if env would contain the following it should work:

[env]
CC=/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
CXX=/Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++

@bilke
Copy link
Contributor

bilke commented Oct 18, 2018

Ups it seems that even the build does not use Xcode 10 but the default one:

line 574: /Applications/Xcode_9.4.1.app/Contents/Developer/usr/bin/make binary
...
line 620: /Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar cq ...
line 621: /Applications/Xcode_9.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib -s ../../../lib/libqtpcre2.a
...

The default one is the one which linked to by /usr/bin/gcc.

@lasote
Copy link
Contributor

lasote commented Oct 18, 2018

To play with different xcodes don't you need to use the xcode-select tool?

@bilke
Copy link
Contributor

bilke commented Oct 18, 2018

Yep you are right, this seems to work: bilke/conan-qt-bincrafters@e2f6e96#diff-fec826feae04e51c0d94076385408bdc

Sorry for bothering..

@lasote
Copy link
Contributor

lasote commented Oct 18, 2018

np!

@lasote lasote closed this as completed Oct 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants