[question] Packaging cross toolchain with relocation and settings_target

[snickl]

What is your question?

Hey, I'm trying to put https://github.com/conan-io/conan/blob/develop2/test/integration/build_requires/test_relocatable_toolchain.py together with https://docs.conan.io/2/examples/cross_build/toolchain_packages.html and just can't seem to make it work.
This is for a buildroot SDK, but stripped down:

base/conanfile.py:

import os

from conan import ConanFile
from conan.tools.files import mkdir


class Base(ConanFile):
    name = "base"
    version = "1.0"
    package_type = "shared-library"
    settings = ("os", "arch", "sdk_build_type")
    _resource_folder = "data"
    no_copy_source = True

    def build(self):
        arch = str(self.settings_target.arch)
        os_lc = str(self.settings_target.os).lower()
        if arch == "armv8":
            arch = "aarch64"
        toolchain_prefix = "-".join([arch, os_lc, "gnu"])
        sdk_build_type = self.settings_target.get_safe("sdk_build_type")
        self.output.info(f"sdk_build_type: {sdk_build_type}")
        sdk_file_name_extra = (
            "-" + sdk_build_type.lower()
            if sdk_build_type and sdk_build_type != "Release"
            else ""
        )
        sdk_file_name = f"{toolchain_prefix}_sdk-buildroot{sdk_file_name_extra}.tar.gz"
        sdk_url = os.path.join("https://artifactserver", sdk_file_name)
        mkdir(self, self._resource_folder)
        self.output.info(f"Retrieving {sdk_url} into {self._resource_folder}")

    def package_id(self):
        self.info.settings_target = self.settings_target
        self.info.settings_target.rm_safe("build_type")
        self.info.settings_target.rm_safe("compiler")

sdk/conanfile.py:

import os

from conan import ConanFile
from conan.tools.files import copy


class Sdk(ConanFile):
    name = "sdk"
    version = "1.0"
    package_type = "shared-library"
    settings = ("os", "arch", "sdk_build_type")
    _resource_folder = "data"
    no_copy_source = True
    build_policy = "missing"
    upload_policy = "skip"

    def build_requirements(self):
        self.tool_requires("base/1.0")

    def configure(self):
        arch = str(self.settings_target.arch)
        os_lc = str(self.settings_target.os).lower()
        if arch == "armv8":
            arch = "aarch64"
        self._toolchain_prefix = "-".join([arch, os_lc, "gnu"])

    def package_id(self):
        self.info.settings_target = self.settings_target
        self.info.settings_target.rm_safe("build_type")
        self.info.settings_target.rm_safe("compiler")

    def package(self):
        base_folder = self.dependencies.build["base"].package_folder
        copy(
            self,
            "*",
            os.path.join(base_folder, self._resource_folder),
            os.path.join(self.package_folder),
        )
        self.output.info(f"Running ./relocate-sdk.sh in cwd={self.package_folder}")

app/conanfile.py:

from conan import ConanFile


class App(ConanFile):
    name = "app"
    version = "1.0"
    package_type = "application"
    settings = ("os", "arch", "compiler", "build_type")

    def build_requirements(self):
        self.tool_requires("sdk/1.0")

    def package_id(self):
        self.info.settings.rm_safe("compiler")

target:

[settings]
os=Linux
arch=armv8
sdk_build_type=Release

workstation:

[settings]
os=Linux
arch=x86_64
sdk_build_type=Release

~/.conan2/settings_user.yml:

sdk_build_type: ["Release", "Debug"]

test.sh:

#!/bin/sh

set -x

conan remove -c "*"
conan create --no-remote --build-require -pr:h=target -pr:b=workstation base/
conan export --no-remote --build-require sdk/
conan create --no-remote -pr:h=target -pr:b=workstation app/
conan list "*#*:*"
conan graph explain --no-remote -pr:h=target -pr:b=workstation app/

Which ends with

======== Closest binaries ========
base/1.0
  base/1.0#8112ee662df1a2967e3b8c12c87d3160%1741642041.4603093 (2025-03-10 21:27:21 UTC)
    base/1.0#8112ee662df1a2967e3b8c12c87d3160:6dade31be6e3e0a864ef0db4f21ac90d653c844f
      remote: Local Cache
      settings: Linux, x86_64, Release
      settings_target: arch=armv8, os=Linux, sdk_build_type=Release
      diff
        settings_target
          expected: arch=x86_64
          existing: arch=armv8
        explanation: This binary was built with different settings_target.

I think the problem is somehow that since the sdk package is built implicitly per dependency, the requested settings (-pr:h=target) don't transfer to settings_target (expected: arch=x86_64).
I've tried more things than bear mentioning and hope you can give me a hint.

The sdk_build_type setting isn't core to this but an adjacent issue, the intent being that the SDK can itself be built in debug mode and I want to be able to select this, but when users of the toolchain apply -s build_type=Debug, that is not what they usually want.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @snickl

Some quick feedback:

https://github.com/conan-io/conan/blob/develop2/test/integration/build_requires/test_relocatable_toolchain.py

This is kind of superseded by the finalize() method, see: https://blog.conan.io/2024/09/03/New-finalize-method.html, that test was there before finalize() was a thing.

then some questions:

• How is it possible that base is a package_type = "shared-library" and also has settings_target? That doesn't make sense, the settings_target only makes sense for applications such as compilers that can create code or binaries intended for a given target platform.
• If base is a shared-library why the sdk has self.tool_requires("base/1.0")? the tool_requires is exclusively for tools executables. For requiring libraries the requires should be used.
• What is the main contents and usage of the sdk? If it is a library-like sdk, then it is possible that instead of self.tool_requires("sdk/1.0") a self.requires("sdk/1.0") is more recommended. The tool_requires() would be more recommended if the sdk is more compiler-like and executables, not as much as libraries

Another thing to have a look would be the vendor=True feature, see https://blog.conan.io/2024/07/09/Introducing-vendoring-packages.html, this is a way to encapsulate dependencies for this kind of re-packaging cases.

[snickl]

Hi @memsharded,
thank you very much for this advice.

I will definitely try the finalize() approach, that looks really promising for getting back to a single, self-contained sdk package. Maybe you could add a hint somewhere in the documentation that cross-toolchain relocation can be managed this way.

What is the main contents and usage of the sdk? If it is a library-like sdk, then it is possible that instead of self.tool_requires("sdk/1.0") a self.requires("sdk/1.0") is more recommended. The tool_requires() would be more recommended if the sdk is more compiler-like and executables, not as much as libraries

I set package_type = "shared-library" because a buildroot or Yocto SDK not only contains the cross compiler, but also a sysroot directory.
Say for example I have the NetworkManager package enabled in my buildroot configuration, and using the SDK want to cross-develop an application for the target system that can talk to the daemon, the headers and libraries to do that are located in the SDK sysroot's include and lib sub-directories.

So an SDK in a way is both a build tool as well as a collection of libraries, for which I would like to propagate CppInfo to Conan packages that use the SDK.
That approach has given me other trouble though and maybe relying on mechanisms like pkgconfig or cmake modules will be better going forward, I'll reconsider.

Another thing to have a look would be the vendor=True feature, see https://blog.conan.io/2024/07/09/Introducing-vendoring-packages.html, this is a way to encapsulate dependencies for this kind of re-packaging cases.

Not sure this fits my use case since it appears to be about Conan packages assimilating other Conan packages. But a SDK sysroot is not comprised of Conan packages, I just manually populate a corresponding [platform_requires].

But thanks again, finalize() really looks like it can help me out in a bunch of ways!