Ask: is it possible to wrap a cross compiler in a conan package?

[ruipires]

After reading the about the build_requires section for the conan profiles, I was under the impression that I could use this to completely wrap a gcc cross compiler in a conan package.

I want to do a proof of concept pushing for internal adoption of conan in my company.
Having this work would add some extra incentives for conan, like:

• smaller checkout sizes (we currently have several cross compilers under source control and they take up a considerable amount of space in our working copies)
• easier to cross compile (the package itself would set the necessary flags / environment variables, so no more sourcing script file to setup the correct cross compiling toolkit).

This being said, I am able to get the cross compiler to work if I have it somewhere on the machine, by configuring a custom profile for it. When I try to move these definitions into a conan package, I always end up having some issues.
Since I cannot refer directly to the paths of the toolkit conan package in the profile file, I need to do the same thing, but use path independent options in the profile flag and adjust the toolkit package to make these work.

I am still unable to make it work, and am wondering if the build_requires feature was meant for something else (not a compiler).

So my question is: is it possible to wrap a compiler in a conan package and use it ? Has this been done ? Are there any examples ?

[memsharded]

Yes, it is possible.
First thing you might want to check are the existing packages for mingw: https://bintray.com/conan-community/conan/mingw_installer%3Aconan

As long as the compiler adds itself to the PATH, typically something like:

def package_info(self):
      self.env_info.PATH = os.path.join(self.package_folder, "bin")

That will be injected into the consumer conanfiles, and used. I have used it successfully in Windows, and I am aware of other users doing similar things for other tools. You can also check the full NDK toolchain example, it is cross-compiling too: http://docs.conan.io/en/latest/howtos/cross_building.html#

Please have a look, try, and report or ask for help if necessary. Thanks!

[lasote]

Take into account that you can declare any environment variable, in the MinGW example I use:

def package_info(self):
    self.env_info.path.append(os.path.join(self.package_folder, "bin"))
    self.env_info.MINGW_HOME = str(self.package_folder)
    self.env_info.CONAN_CMAKE_GENERATOR = "MinGW Makefiles"
    self.env_info.CXX = os.path.join(self.package_folder, "bin", "g++.exe")
    self.env_info.CC = os.path.join(self.package_folder, "bin", "gcc.exe")

[ruipires]

First of all, thank you all for your quick responses.

I am still having some issues, and am not sure why.

I am testing my package like this, after fiddling with the package_info method in my edit-test cycle:

conan export rui/test
conan test_package rui/test  --profile qoriq50 -t

I do this because I want to avoid downloading the toolkit again (it is really big), which I do in the source method.

In the test I just read CC and CFLAGS from the environment and invoke the compiler via self.run directly (avoiding any cmake or other helpers for now, to simplify the scenario).

I am using:

    def package_info(self):
        self.env_info.CC = os.path.join(self.package_folder, "build/rootfs/sdk/sysroots/x86_64-wrlinuxsdk-linux/usr/toolchain/bin/powerpc-wrs-linux-gnu-gcc")
        self.env_info.CFLAGS = "-g -mcpu=e500mc -O2 -fno-omit-frame-pointer -pipe -m32 -DARCH=powerpc --sysroot=%s" % os.path.join(self.package_folder, "build/rootfs/sdk/sysroots/tlabqoriqsmall-wrs-linux")

And my qoriq50 profile is very spartan:

[settings]
os=Linux
compiler=gcc
compiler.version=4.6
compiler.libcxx=libstdc++
arch=ppc
[build_requires]
qoriq50/1.0@rui/test

Note that I did not define a CC in this profile, leaving that to be defined by the package_info method.

Now, if I run, it fails with:

ERROR: PROJECT: Error in build() method, line 18
        self.run(cmdline)
        ConanException: Error 32512 while executing /usr/bin/gccBLARG -g -mcpu=e500mc -O2 -fno-omit-frame-pointer -pipe -m32 -DARCH=powerpc --sysroot=/home/vagrant/.conan/data/qoriq50/1.0/rui/test/package/bc0b17cabac44b49a4eac714e6b2e466085b96ee/build/rootfs/sdk/sysroots/tlabqoriqsmall-wrs-linux /home/vagrant/teste.c -o teste

Notice the weird /usr/bin/gccBLARG ...
I changed the CC in the default profile (which I am not using here) to that to figure out what was happening.
Somehow, the default profile is being used (wasn't expecting that), and the CC I defined in package_info is no longer present.

Since some of the variables defined in package_info are present in the environment (CFLAGS, for example), this is really confusing me.

Do profiles have a cascading behavior ?

p.s.:

$> conan --version
Conan version 0.27.0

[memsharded]

You have:

$ conan create user/testing --keep-source

To avoid retrieving again the source code. If you want also to skip the export step, then you can also use --not-export

The error is weird, indeed. Can I see your full build() method? It could be also useful that you printed the os.getenv("CC") and the full cmdLine variable in such build method.

[ruipires]

Here are the remaining portions of my conanfile.py:

class Qoriq50Conan(ConanFile):
    name = "qoriq50"
    url = ...
    settings = "os", "compiler", "arch"
    exports = "*.cmake"

    def configure(self):
        if self.settings.os != "Linux":
            raise Exception("Only Linux supported")
        if str(self.settings.compiler) != "gcc":
            raise Exception("Compiler not supported, only gcc available")
        if str(self.settings.arch) != "ppc":
            raise Exception("Architecture not supported, only ppc available")

        if not tools.OSInfo().is_linux:
            raise Exception("Host operating system not supported")

    def source(self):
        self.output.info("Checking out %s" % self.url)
        self.run("svn co %s" % self.url)

        self.output.info("Pruning broken links")
        self.run("cd tlabqoriq50_small && find -xtype l -delete")

        self.output.info("Removing outdated cmake from toolkit")
        os.remove("tlabqoriq50_small/build/rootfs/sdk/sysroots/x86_64-wrlinuxsdk-linux/usr/bin/cmake")

    def package(self):
        self.copy(pattern="*", src="tlabqoriq50_small", keep_path=True)
        self.copy("powerpc-wrs-linux-gnu-target.cmake")

[memsharded]

no build() method?

[ruipires]

Not on the main conanfile.py, only on the test_package:

class QorIQ50TestConan(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = "cmake"

    def build(self):
        self.run("env | grep -i CC")
        self.run("env | grep -i CFLAGS")

        self.output.info("Compiling manually")
        cmdline = "%s %s /home/vagrant/teste.c -o teste" % ( os.environ["CC"],  os.environ["CFLAGS"])
        self.output.info("> %s" % cmdline)
        self.run(cmdline)

    def test(self):
        os.chdir("bin")
        self.run("file .%steste | grep PowerPC" % os.sep)
        self.output.info("file 'teste' should be described as a PowerPC executable")

[memsharded]

And the output now with those commands? :)

Please use the conan create command and paste the full output here. Thanks!

[ruipires]

Reduced this as much as possible, you can see the minimal setup in this gist.

Running conan test_package rui/test --profile toolkit I see this output:

...
toolkit/1.0@rui/test test package: Running build()
CC=/usr/bin/gccBLARG
CFLAGS=custom cflags
toolkit/1.0@rui/test test package: > /usr/bin/gccBLARG custom cflags
sh: 1: /usr/bin/gccBLARG: not found
ERROR: PROJECT: Error in build() method, line 12
        self.run(cmdline)
        ConanException: Error 32512 while executing /usr/bin/gccBLARG custom cflags

the /usr/bin/gccBLARG is defined in my default profile:

[build_requires]
[settings]
os=Linux
arch=x86_64
compiler=gcc
compiler.version=6.3
compiler.libcxx=libstdc++11
build_type=Release
[options]
[scopes]
[env]
CC=/usr/bin/gccBLARG
CXX=/usr/bin/g++

[ruipires]

I think I misused the --keep-source flag.
It is downloading from sources. I will take a while before I can show the result of this command.

$> conan create rui/test --profile qoriq50 --keep-source
qoriq50/1.0@rui/test: Exporting package recipe
WARN: Conanfile doesn't have 'description'.
It is recommended to add it as attribute
qoriq50/1.0@rui/test: The stored package has not changed
qoriq50/1.0@rui/test: Installing package
Requirements
    qoriq50/1.0@rui/test from local
Packages
    qoriq50/1.0@rui/test:bc0b17cabac44b49a4eac714e6b2e466085b96ee

qoriq50/1.0@rui/test: WARN: Forced build from source
qoriq50/1.0@rui/test: Installing build requirements of: qoriq50/1.0@rui/test
qoriq50/1.0@rui/test: Build requires: [qoriq50/1.0@rui/test]
qoriq50/1.0@rui/test: Requirements
qoriq50/1.0@rui/test:     qoriq50/1.0@rui/test from local
qoriq50/1.0@rui/test: Packages
qoriq50/1.0@rui/test:     qoriq50/1.0@rui/test:bc0b17cabac44b49a4eac714e6b2e466085b96ee
qoriq50/1.0@rui/test:
qoriq50/1.0@rui/test: WARN: Forced build from source
qoriq50/1.0@rui/test: Building your package in /home/vagrant/.conan/data/qoriq50/1.0/rui/test/build/bc0b17cabac44b49a4eac714e6b2e466085b96ee
qoriq50/1.0@rui/test: Configuring sources in /home/vagrant/.conan/data/qoriq50/1.0/rui/test/source
qoriq50/1.0@rui/test: Checking out https://.../tlabqoriq50_small
A    tlabqoriq50_small/kernelModules
...

[memsharded]

Ok, now I get it. You are totally right, this is an issue. Let me explain what is happening:

• User input, in this case, profiles, always have precedence over requirements, so user is always in control.
• The profile specified is accumulated to the default profile.

While the second usually works fine for other things, I get this is a problem. We are discussing to see if it is possible to improve this. In the meantime, please remove the CC from the default profile, only use it in other profiles.

[ruipires]

OK, its clear now what was happening.
Thanks.

[memsharded]

We will be removing this behavior in next release 0.28. Profiles won't automatically extend the "default" profile, they will have to explicitely include it if they want.