flatbuffers: refactoring

[SpaceIm]

• deprecate flatc, flatbuffers and options_from_context options (there was an attempt in [flatbuffers] remove misleading option #4863). All these options were part of a counterintuitive mechanism preventing to build the lib in build context and the executable in host context.
  Instead:
  • flatc is always built now, as well as flatbuffers lib (either static, shared or header only).
  • we provide flatbuffers::flatc target (like upstream config file), and handle executable discovery from build context if cross-building with 2 profiles (it was managed by the weird options_from_context before), or host context if native build. It's a more elegant solution than Flatbuffers flatc integration #3345, and similar to the trick used in protobuf and grpc recipes.
• relocatable shared lib on macOS: see [conan-center] Add a hook to check whether installed shared libs are relocatable on Linux & macOS hooks#376
• CMakeDeps support
• fix Find module name (it's FindFlatBuffers.cmake, while config file is FlatbuffersConfig.cmake): see https://github.com/google/flatbuffers/tree/v2.0.0/CMake
• always rely on upstream CMakeLists to build and package flatbuffers (event for header-only) => less branching in the recipe.
• only provide functions of BuildFlatBuffers.cmake through find_package() (ie cmake_find_package/cmake_find_package_multi/CMakeDeps). At some point it doesn't make sense and it's not worth the effort to support consumers trying to call these functions without calling find_package(Flatbuffers) before.
  It's worth noting that this module file & its functions are not part of upstream config file, only Find module file. I've added this file to both, but I don't recommend to rely on these functions.

/cc @jgsogo @mjvankampen

closes #5616
closes #8374

---

• I've read the guidelines for contributing.
• I've followed the PEP8 style guides for Python code in the recipes.
• I've used the latest Conan client version.
• I've tried at least one configuration locally with the
  conan-center hook activated.

[mjvankampen]

By building flatc always I cannot use this recipe anymore for platforms where flatc cannot be build. I'm not sure if there are still platforms that do not support flatc but do support flatbuffers. This is not such a weird situation, there are multiple platforms where the compiler/code generator is more restrictive that the produced code. This was the original reason for the weird option_from_context as it was the only way I could find to include flatc in the build context but not in the host context.

[SpaceIm]

By building flatc always I cannot use this recipe anymore for platforms where flatc cannot be build.

Which platforms? If there are platforms not supporting flatc executable, I can keep flatc option, but not option_from_context nor flatbuffers.

[mjvankampen]

QNX for example. Not sure about others. I don't think you can set the flatc option one way for the build context and a different for the host context. Or is this possible now?

[SpaceIm]

QNX for example. Not sure about others. I don't think you can set the flatc option one way for the build context and a different for the host context. Or is this possible now?

If I add flatc option again, it would be possible:

conanfile.txt

[requires]
flatbuffers/2.0.0
[build_requires]
flatbuffers/2.0.0

conan install conanfile.txt -o:h flatbuffers:flatc=False -o:b flatbuffers:flatc=True -pr:h host_profile -pr:b build_profile -b missing

EDIT: actually, -o:b option seems to be broken conan-io/conan#10425 (comment)

What is the error on QNX?

[SpaceIm]

ERROR: Missing binary: flatbuffers/2.0.0:17c6b956e973041a99732b9f33647b34bae851a9
flatbuffers/2.0.0: WARN: Can't find a 'flatbuffers/2.0.0' package for the specified settings, options and dependencies:
- Settings: arch=x86_64, build_type=Release, compiler=apple-clang, compiler.libcxx=libc++, compiler.version=12.0, os=Macos
- Options: flatbuffers=deprecated, flatc=deprecated, header_only=True, options_from_context=deprecated
- Dependencies: 
- Requirements: 
- Package ID: 17c6b956e973041a99732b9f33647b34bae851a9

ERROR: Missing prebuilt package for 'flatbuffers/2.0.0'
Use 'conan search flatbuffers/2.0.0 --table=table.html -r=remote' and open the table.html file to see available packages
Or try to build locally from sources with '--build=flatbuffers'

@jgsogo what is exactly the command executed in CI? This error should never happen with conan create when we add the same dependency in build requirements of test package, even if package id of build context is not available.
It seems that other conan create commands succeeded for armv8 without missing prebuilt package, except when header_only=True.

[jgsogo]

Hi! The error comes from the conan test command. The test_package/conanfile.py is adding flatbuffers to build context (as a build-requires), and this is something we can't do although it might succeed under some circumstances (see 1️⃣ below).

When building for Apple M1, the command is conan test ... --profile:host=<the-host-profile> --profile:build=default (two profiles), the profile:host matches the one used to build the package, but the profile:build is different:

Configuration (profile_host):
[settings]
arch=armv8
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=12.0
os=Macos
[options]
flatbuffers:header_only=True
[build_requires]
[env]

Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=12.0
os=Macos
os_build=Macos
[options]
[build_requires]
[env]

And the packageID computed for each context are different:

Requirements
    flatbuffers/2.0.0 from local cache - Cache
Packages
    flatbuffers/2.0.0:aa8bfbbddca8feb09a7119f3e659ee3aa3a1d452 - Download
Build requirements
    flatbuffers/2.0.0 from local cache - Cache
Build requirements packages
    flatbuffers/2.0.0:17c6b956e973041a99732b9f33647b34bae851a9 - Missing

Of course, the packageID for the build context is missing because it hasn't been built. When creating the package in the previous step we build the one for the host context.

We cannot test in test_package/conanfile.py any other configuration than the one that has been built. We cannot add the package as a build-requires, at least in Conan v1 that doesn't know how to rotate the profiles.

---

1️⃣ It can succeed if the package for the build context has already been built by another thread and uploaded to the remote... typically, it will succeed if we retrigger the build one more time for the same recipe revision.

This is probably the reason why other packages have succeeded. Having a look at the logs, they are using two different packageIDs:

Requirements
    flatbuffers/2.0.0 from local cache - Cache
Packages
    flatbuffers/2.0.0:c36572cdc38be7dbc966c1f9dd7cc86c518bcb6c - Download
Build requirements
    flatbuffers/2.0.0 from local cache - Cache
Build requirements packages
    flatbuffers/2.0.0:6a83d7f783e7ee89a83cf2fe72b5f5f67538e2a6 - Download

and if they work, it is just because that packageID has been generated before by a different job.

Of course we cannot rely on this behavior.

[jgsogo]

btw, thanks a lot for taking care of this recipe, all those fixes were really needed 🎉

@mjvankampen , if flatc cannot be built for some configuration (and given the bug you are mentioning), we can use a different approach and remove the option or change its default value for that configuration. It's something we have done before in some recipes and helps us to deal with this kind of situation.

[SpaceIm]

and if they work, it is just because that packageID has been generated before by a different job.

Of course we cannot rely on this behavior.

Thanks for the detailed explanation.

So what should I do here?

• close/open and hope that package id of build requirement have been uploaded previously?
• disable test if cross-build? (but I really want to test that we can call flatbuffers::flatc target from build context).

btw, thanks a lot for taking care of this recipe, all those fixes were really needed 🎉

@mjvankampen , if flatc cannot be built for some configuration (and given the bug you are mentioning), we can use a different approach and remove the option or change its default value for that configuration. It's something we have done before in some recipes and helps us to deal with this kind of situation.

QNX is not referenced in default settings.yml, so there is no way to add a reliable branching based on this OS I think.

I guess something like this would be fine, but it's weird to reference an OS id not even supported in settings.yml

config_options(self):
    if str(self.settings.os).upper() == "QNX":
        self.options.flatc = False

[jgsogo]

• disable test if cross-build? (but I really want to test that we can call flatbuffers::flatc target from build context).

Indeed, we can only test the build context if both profiles (host and build) are the same (same packageID). So far this only happens (atm in CCI) in scenarios other than cross-building ones.

For the future, Conan should provide a way to indicate that the test_package/conanfile.py is testing the package as build_requires (there was a test_type for it) and then Conan should use the input --profile:host to feed the build context (ensure we are testing the package we have just build)... and probably use the same one as the profile for the target context (or just accept one more parameter). If rotating the profiles this way sound a bit weird, think about the conan create --profile:host --profile:build command where it is clear that the package built (host context) should be used as a build-requires in the testing step, so Conan needs to rotate the profiles.

[SpaceIm]

btw, thanks a lot for taking care of this recipe, all those fixes were really needed 🎉

@mjvankampen , if flatc cannot be built for some configuration (and given the bug you are mentioning), we can use a different approach and remove the option or change its default value for that configuration. It's something we have done before in some recipes and helps us to deal with this kind of situation.

There is a serious issue in 2 profiles I think. Even with this trick, if someone has flatbuffers in requirements (for the library) and build requirement (for flatc), and flatc option is False in requirements only, it won't work because build requirement will inherit flatbuffers:flatc=False (and it should not obviously): conan-io/conan#10425 (comment)

[SpaceIm]

@mjvankampen would it be fine with 8b99a52?

[conan-center-bot]

All green in build 4 (267544fdbf5c51bc9c4349a691c606a442fa9546):

• flatbuffers/2.0.0@:
  All packages built successfully! (All logs)

• flatbuffers/2.0.5@:
  All packages built successfully! (All logs)

• flatbuffers/1.12.0@:
  All packages built successfully! (All logs)

• flatbuffers/1.11.0@:
  All packages built successfully! (All logs)

[SSE4]

let's move this one forward @SpaceIm . there are serious issues with 2 profiles in v1, but nothing we can do about it in CCI right now :(
I don't think it should block your PR, as it's an improvement on its own.

[qh-huang]

Hi @SpaceIm ,
I just updated my conan to 1.49.0 and suddenly the function build_flatbuffers() in my CMakeList.txt not working anymore. I think it's related to this patch.

After reading this discussion loop, I still got following 2 questions.

1. I can see the cmake function build_flatbuffers() is in BuildFlatBuffers.cmake which is provided in official conan flatbuffers package, but how should I use it in my CMakeList.txt?
2. How to work with host executable with conan? I've tried to adopt the best practice (i.e. use the flatc executable of host machine, for me which is ubuntu 20.04, and use find_package(Flatbuffers) to import the cmake variables. However, cmake complains as following:

CMake Error at CMakeLists.txt:57 (find_package):
  By not providing "FindFlatbuffers.cmake" in CMAKE_MODULE_PATH this project
  has asked CMake to find a package configuration file provided by
  "Flatbuffers", but CMake did not find one.

  Could not find a package configuration file provided by "Flatbuffers" with
  any of the following names:

    FlatbuffersConfig.cmake
    flatbuffers-config.cmake

Thanks.

[elejke]

Have the same problem on linux as @qiao-tw when building onnxruntime/1.7.1 package with flatbuffers/1.12.0

what is the current solution for this?