Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic Conan support #401

Merged
merged 52 commits into from
Apr 4, 2022
Merged

Add basic Conan support #401

merged 52 commits into from
Apr 4, 2022

Conversation

PengZheng
Copy link
Contributor

@PengZheng PengZheng commented Mar 15, 2022

Now we support Conan package manager, and the modular structure of cmake project is improved along the way:

  1. Replace the broken automatic dependency deduction in celix_subproject with automatic error detection. Use celix_subproject systematically to reflect the current modular structure, and thus allowing more to be opted out. Essential and non-essential usages are carefully differentiated: e.g., PUBSUB_PSA_WS uses HTTP_ADMIN essentially, while all bundles use log_helper non-essentially. Now it's possible to turn off all bundles.
  2. Minor cmake style improvements, including: replacing add_dependencies with add_celix_bundle_dependencies; use CMAKE_INSTALL_INCLUDEDIR instead of fixed include; replacing $<INSTALL_INTERFACE:> with INCLUDES DESTINATION.
  3. conanfile.py is also updated according to the current project structure. The default options reflect a reasonably minimal and functionally stable configuration: framework without etcd, log_admin, http_admin, pubsub_pas_ws with no need of external discovery mechanism, rsa with preconfigured discovery.

Now test_package covers all stable functionalities. This PR is finished, and is ready for review. @pnoltes
This PR may seems a little scary at a first glance, considering its size. However, I shall emphasize that this PR does not change fundamentally how Celix builds, and that CMake build system can be used without conan as before. Though more celix_subprojects are used, it should change nothing with a default ON .

It used GPLv3ed zmq by default, which cannot be included in most proprietary softwares deployed in embedded device.
There are still some rough edges in CMake in-project dependency management.
Use find_dependency without REQUIRED to handle find_package(Celix) correctly (i.e., use Celix as an optional package).
Avoid redefining libzip::libzip and support Jansson target for etcdlib.
…rdingly.

Replace the broken automatic dependency deduction in celix_subproject with automatic error detection. Use celix_subproject systematically to reflect the current modular structure, and thus allowing more to be opted out. Essential and non-essential usages are carefully differentiated: e.g., PUBSUB_PSA_WS uses HTTP_ADMIN essentially, while all bundles use log_helper non-essentially. Now it's possible to turn off all bundles.

Minor cmake style improvements, including: replacing add_dependencies with add_celix_bundle_dependencies; use ${CMAKE_INSTALL_INCLUDEDIR} instead of fixed `include`; replacing $<INSTALL_INTERFACE:> with INCLUDES DESTINATION.

conanfile.py is also updated according to the current project structure. The default options reflect a reasonably minimal and functionally stable configuration: framework without etcd, log_admin, http_admin, pubsub_pas_ws with no need of external discovery mechanism, rsa with preconfigured discovery.
@@ -908,8 +908,9 @@ function(install_celix_bundle_targets)

get_target_property(EXPORT_BUNDLES celix-bundles EXPORT_${EXPORT_NAME}_BUNDLES)

if (NOT DEFINED EXPORT_BUNDLES)
message(FATAL_ERROR "Export ${EXPORT_NAME} not defined. Did you forgot to use a install_celix_bundle with the 'EXPORT ${EXPORT_NAME}' option?")
if (NOT DEFINED EXPORT_BUNDLES OR NOT EXPORT_BUNDLES)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that it is possible to turn off all bundles, this is not FATAL error any more.

set(CELIX_MAJOR "2")
set(CELIX_MINOR "2")
set(CELIX_MICRO "1")
if (NOT DEFINED CELIX_MAJOR)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if not defined externally (by conan)

@@ -15,7 +15,7 @@
# specific language governing permissions and limitations
# under the License.

celix_subproject(DEPLOYMENT_ADMIN "Option to enable building the Deployment Admin Service bundles" ON DEPS framework launcher shell_tui log_writer)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Framework is always built. shell_tui and log_writer are only used for the demo application, thus are of non-essential usage.

@PengZheng
Copy link
Contributor Author

test_package for three components added, more to come later.

@codecov-commenter
Copy link

codecov-commenter commented Mar 16, 2022

Codecov Report

Merging #401 (7c68393) into master (6f1817d) will decrease coverage by 1.35%.
The diff coverage is 90.90%.

@@            Coverage Diff             @@
##           master     #401      +/-   ##
==========================================
- Coverage   73.13%   71.78%   -1.36%     
==========================================
  Files         205      180      -25     
  Lines       31274    29281    -1993     
==========================================
- Hits        22872    21018    -1854     
+ Misses       8402     8263     -139     
Impacted Files Coverage Δ
...ibs/pushstreams/api/celix/impl/PushEventConsumer.h 100.00% <ø> (ø)
libs/utils/src/array_list.c 87.57% <90.90%> (ø)
...les/pubsub/pubsub_admin_zmq/src/pubsub_zmq_admin.c 52.96% <0.00%> (-13.93%) ⬇️
libs/framework/include/celix/dm/Component_Impl.h 82.81% <0.00%> (-10.53%) ⬇️
...ibs/framework/include/celix/dm/ServiceDependency.h 92.85% <0.00%> (-7.15%) ⬇️
...b/pubsub_admin_zmq/src/pubsub_zmq_topic_receiver.c 76.50% <0.00%> (-4.82%) ⬇️
...sub/pubsub_admin_zmq/src/pubsub_zmq_topic_sender.c 84.38% <0.00%> (-2.54%) ⬇️
libs/utils/src/hash_map.c 93.25% <0.00%> (-1.69%) ⬇️
...ramework/include/celix/dm/ServiceDependency_Impl.h 93.70% <0.00%> (-1.63%) ⬇️
bundles/pubsub/pubsub_utils/src/pubsub_utils.c 68.88% <0.00%> (-1.12%) ⬇️
... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f1817d...7c68393. Read the comment docs.

if self.options.enable_testing:
self.options['gtest'].shared = True
if self.options.enable_address_sanitizer:
self.options["cpputest"].with_leak_detection = False
Copy link
Contributor Author

@PengZheng PengZheng Mar 27, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without Leak Detection Plugin, clang address sanitizer now works with cpputest, the following is addressed:

sanitizer: false #note sanitizer on clang with cpputest does not work

We cannot modify distribution's cpputest package, but we can easily do so in Conan's virtual environment.

set(CMAKE_C_FLAGS "-fsanitize=address -fno-omit-frame-pointer ${CMAKE_C_FLAGS}")
set(CMAKE_CXX_FLAGS "-fsanitize=address -fno-omit-frame-pointer ${CMAKE_CXX_FLAGS}")
set(CMAKE_C_FLAGS "-DCPPUTEST_MEM_LEAK_DETECTION_DISABLED -fsanitize=address -fno-omit-frame-pointer ${CMAKE_C_FLAGS}")
set(CMAKE_CXX_FLAGS "-DCPPUTEST_MEM_LEAK_DETECTION_DISABLED -fsanitize=address -fno-omit-frame-pointer ${CMAKE_CXX_FLAGS}")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When enabling ASAN, it makes no sense to use the inferior CPPUTest memory leak detection.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree

@@ -135,7 +135,7 @@ function (setup_target_for_coverage)
# Capturing lcov counters and generating report
COMMAND ${CMAKE_COMMAND} -E make_directory ${CMAKE_BINARY_DIR}/coverage
COMMAND ${LCOV_PATH} --directory ${COVERAGE_SCAN_DIR} --capture --output-file ${OUTPUT_FILE}
COMMAND ${LCOV_PATH} --remove ${OUTPUT_FILE} '**/mock/*' '**/test/*' '**/gtest/*' '**/tst/*' '**/celix/gen/*' '**/googletest_project/*' '**/glog/*' '/usr/*' --output-file ${OUTPUT_FILE}.cleaned
COMMAND ${LCOV_PATH} --remove ${OUTPUT_FILE} '**/mock/*' '**/.conan/*' '**/test/*' '**/gtest/*' '**/tst/*' '**/celix/gen/*' '**/googletest_project/*' '**/glog/*' '/usr/*' --output-file ${OUTPUT_FILE}.cleaned
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To remove all external dependencies.

@pnoltes
Copy link
Contributor

pnoltes commented Apr 3, 2022

@PengZheng: I wanted to create a separate PR with some changes to this PR, but I accidentally pushed directly on this PR.

@pnoltes
Copy link
Contributor

pnoltes commented Apr 3, 2022

I started testing the Celix Conan setup on a macOS system and something did not work:

  • conan install failed, because both cmake 3.17.5 and libzip 1.7.3 needed an different openssl version (1.1.1m and 1.1.1n). Currently I solved this by explicitly adding a openssl dep, but maybe this can be solved cleaner
    -Wl,--unresolved-symbols=ignore-in-shared-libs does not work on macOS, so conanfile now does this behind an if
  • For me self.version was not so, so made this optional
  • The conan generated Findjansson creates a jansson::jansson imported target and not a Jansson imported target which was generated by Celix's FIndJansson.cmake. This has now been updated to jansson::jansson

I still have some troubles with running the Celix tests and I will look into this a bit later.

@PengZheng
Copy link
Contributor Author

PengZheng commented Apr 3, 2022

I wanted to create a separate PR with some changes to this PR, but I accidentally pushed directly on this PR.

No problem. Let's address remaining issues together.
I answered part of your questions in #408

@pnoltes
Copy link
Contributor

pnoltes commented Apr 3, 2022

I wanted to create a separate PR with some changes to this PR, but I accidentally pushed directly on this PR.

No problem. Let's address remaining issues together. I answered part of your questions in #408

This is the error I am getting if openssl is not fixed to 1.1.1n:

ERROR: Conflict in cmake/3.17.5:
    'cmake/3.17.5' requires 'openssl/1.1.1n' while 'libzip/1.8.0' requires 'openssl/1.1.1m'.
    To fix this conflict you need to override the package 'openssl' in your root package.
ERROR: conanbuildinfo.txt file not found in /Users/pnoltes/workspace/celix/cmake-build-debug
It is required for this command
You can generate it using 'conan install'

I tried changing the version of cmake and libzip, but I could not find version combination that does not raise this issue.

@pnoltes
Copy link
Contributor

pnoltes commented Apr 3, 2022

When using conan create with a profile for build and host, I get the same error:

% conan create --build-require -pr:b default -pr:h default . celix/2.2.3@zhengpeng/testing
Exporting package recipe
celix/2.2.3@zhengpeng/testing exports_sources: Copied 161 '.txt' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 41 '.md' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 426 '.h' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 286 '.c' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 99 '.cc' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 4 '.html' files: index.html, index.html, index.html, index.html
celix/2.2.3@zhengpeng/testing exports_sources: Copied 4 '.js' files: ansi_up.js, script.js, script.js, script.js
celix/2.2.3@zhengpeng/testing exports_sources: Copied 17 '.in' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 34 '.cpp' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 44 '.descriptor' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 6 '.json' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 7 '.avpr' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 2 '.xml' files: desc.xml, desc.xml
celix/2.2.3@zhengpeng/testing exports_sources: Copied 5 '.inl' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 11 '.properties' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 6 files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 4 '.adoc' files: pubsub_serialization.adoc, async_remote_admin.adoc, configured_discovery.adoc, pushstreams.adoc
celix/2.2.3@zhengpeng/testing exports_sources: Copied 25 '.cmake' files
celix/2.2.3@zhengpeng/testing exports_sources: Copied 1 '.sh' file: create_target_filesystem.sh
celix/2.2.3@zhengpeng/testing exports_sources: Copied 1 '.py' file: conanfile.py
celix/2.2.3@zhengpeng/testing exports_sources: Copied 2 '.avdl' files: simple.avdl, complex.avdl
celix/2.2.3@zhengpeng/testing exports_sources: Copied 1 '.png' file: statediagram.png
celix/2.2.3@zhengpeng/testing exports_sources: Copied 1 '.Android' file: Dockerfile.Android
celix/2.2.3@zhengpeng/testing exports_sources: Copied 3 '.MF' files: MANIFEST.MF, MANIFEST.MF, MANIFEST.MF
celix/2.2.3@zhengpeng/testing: The stored package has not changed
celix/2.2.3@zhengpeng/testing: Exported revision: 203b6bd856d78da272db6ab3c6076d83
Configuration (profile_host):
[settings]
arch=armv8
arch_build=armv8
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=13
os=Macos
os_build=Macos
[options]
[build_requires]
[env]

Configuration (profile_build):
[settings]
arch=armv8
arch_build=armv8
build_type=Release
compiler=apple-clang
compiler.libcxx=libc++
compiler.version=13
os=Macos
os_build=Macos
[options]
[build_requires]
[env]

WARN: libcurl/7.80.0: requirement zlib/1.2.11 overridden by celix/2.2.3@zhengpeng/testing to zlib/1.2.12 
WARN: libzip/1.8.0: requirement zlib/1.2.11 overridden by celix/2.2.3@zhengpeng/testing to zlib/1.2.12 
WARN: libxml2/2.9.13: requirement zlib/1.2.11 overridden by celix/2.2.3@zhengpeng/testing to zlib/1.2.12 
jansson/2.14: Main binary package '87dfe75185c1ec4d4a45265bdca4ccc582ee08d9' missing. Using compatible package '9318e285a716ea373adae1284f599e416da5d17d'
libffi/3.4.2: Main binary package '9626d29b9c4fd933cf8b22e708bd74cf979bc636' missing. Using compatible package '8e74bc12f7afe2971bedf38d3cbd134313ec2dbb'
libiconv/1.16: Main binary package 'e98cc99c78f8ae352930d807a85fc7d6df891023' missing. Using compatible package '095512ed878f14a09dd732e9f6868729dd437529'
openssl/1.1.1m: Main binary package '9626d29b9c4fd933cf8b22e708bd74cf979bc636' missing. Using compatible package '8e74bc12f7afe2971bedf38d3cbd134313ec2dbb'
xz_utils/5.2.5: Main binary package 'e98cc99c78f8ae352930d807a85fc7d6df891023' missing. Using compatible package '095512ed878f14a09dd732e9f6868729dd437529'
zlib/1.2.12: Main binary package '9626d29b9c4fd933cf8b22e708bd74cf979bc636' missing. Using compatible package '8e74bc12f7afe2971bedf38d3cbd134313ec2dbb'
zstd/1.5.2: Main binary package '730f6fa15d7dd036e45ab9c2ddeba72c3a5b2670' missing. Using compatible package '29c13c3349c6850cc905f19384cd71df20f5ae02'
libxml2/2.9.13: Main binary package '85f3a088c51eae9cf361c968abcbafddfb81b2dd' missing. Using compatible package '450a7ddd4356170be96db326460ee375171471f5'
celix/2.2.3@zhengpeng/testing: Forced build from source
ERROR: Conflict in cmake/3.17.5:
    'cmake/3.17.5' requires 'openssl/1.1.1n' while 'libzip/1.8.0' requires 'openssl/1.1.1m'.
    To fix this conflict you need to override the package 'openssl' in your root package.
    

This implies that the build tools and build requirements are not in different contexts.
The conan version I am using is 1.47.0

@pnoltes
Copy link
Contributor

pnoltes commented Apr 3, 2022

@PengZheng I reverted the changes to version. For my system the fixed openssl is still needed.
I also have some troubles getting this to work nicely with CLion, but IMO that is something that can be tackled later.

@PengZheng
Copy link
Contributor Author

conan create --build-require -pr:b default -pr:h default . celix/2.2.3@zhengpeng/testing

Using the command line you provided, I successfully reproduced the problem. The root cause is --build-require.
Removing it, conan create -pr:b default -pr:h default . celix/2.2.3@zhengpeng/testing runs successfully. You can see openssl actually appears twice in different contexts:

Version ranges solved
    libffi/* versions found in 'conan-center' remote
    Version range '>=3.2.1 <4.0.0' required by 'celix/2.2.3@zhengpeng/testing' resolved to 'libffi/3.4.2' in remote 'conan-center'
    jansson/* versions found in 'conan-center' remote
    Version range '>=2.12 <3.0.0' required by 'celix/2.2.3@zhengpeng/testing' resolved to 'jansson/2.14' in remote 'conan-center'
    libcurl/* versions found in 'conan-center' remote
    Version range '>=7.64.1 <8.0.0' required by 'celix/2.2.3@zhengpeng/testing' resolved to 'libcurl/7.80.0' in remote 'conan-center'
    zlib/* versions found in 'conan-center' remote
    Version range '>=1.2.8 <2.0.0' required by 'celix/2.2.3@zhengpeng/testing' resolved to 'zlib/1.2.12' in remote 'conan-center'
    libzip/* versions found in 'conan-center' remote
    Version range '>=1.7.3 <2.0.0' required by 'celix/2.2.3@zhengpeng/testing' resolved to 'libzip/1.8.0' in remote 'conan-center'
    libxml2/* versions found in 'conan-center' remote
    Version range '>=2.9.9 <3.0.0' required by 'celix/2.2.3@zhengpeng/testing' resolved to 'libxml2/2.9.13' in remote 'conan-center'

Installing package: celix/2.2.3@zhengpeng/testing
Requirements
    bzip2/1.0.8 from 'conan-center' - Downloaded
    celix/2.2.3@zhengpeng/testing from local cache - Cache
    jansson/2.14 from 'conan-center' - Downloaded
    libcurl/7.80.0 from 'conan-center' - Downloaded
    libffi/3.4.2 from 'conan-center' - Downloaded
    libiconv/1.16 from 'conan-center' - Downloaded
    libuuid/1.0.3 from 'conan-center' - Downloaded
    libxml2/2.9.13 from 'conan-center' - Downloaded
    libzip/1.8.0 from 'conan-center' - Downloaded
    openssl/1.1.1m from 'conan-center' - Downloaded
    xz_utils/5.2.5 from 'conan-center' - Downloaded
    zlib/1.2.12 from 'conan-center' - Downloaded
    zstd/1.5.2 from 'conan-center' - Downloaded
Packages
    bzip2/1.0.8:da606cf731e334010b0bf6e85a2a6f891b9f36b0 - Download
    celix/2.2.3@zhengpeng/testing:3a976adf4af022727b766e72a23112f5d1842d80 - Build
    jansson/2.14:c47bb6db96f23b2ee1060fcd1f35bc24e9898635 - Download
    libcurl/7.80.0:8d6549fec3f9898f66162c40cf7d0678a9f79c29 - Download
    libffi/3.4.2:2af715f34a7c8c2aeae57b25be0a52c4110dc502 - Download
    libiconv/1.16:6af9cc7cb931c5ad942174fd7838eb655717c709 - Download
    libuuid/1.0.3:2af715f34a7c8c2aeae57b25be0a52c4110dc502 - Download
    libxml2/2.9.13:988c12d94ac6fc710966303867aa6edf51330969 - Download
    libzip/1.8.0:a4f3806664341dffa1db6ce9f085d0fb1edb2b7a - Download
    openssl/1.1.1m:2af715f34a7c8c2aeae57b25be0a52c4110dc502 - Download
    xz_utils/5.2.5:6af9cc7cb931c5ad942174fd7838eb655717c709 - Download
    zlib/1.2.12:2af715f34a7c8c2aeae57b25be0a52c4110dc502 - Download
    zstd/1.5.2:a1fcff3227a663d62f35b4eedde78f4ce0fcd438 - Download
Build requirements
    cmake/3.17.5 from 'conan-center' - Downloaded
    openssl/1.1.1n from 'conan-center' - Downloaded
Build requirements packages
    cmake/3.17.5:5c09c752508b674ca5cb1f2d327b5a2d582866c8 - Download
    openssl/1.1.1n:2af715f34a7c8c2aeae57b25be0a52c4110dc502 - Download

If you want to build missing requirement, the command line should be conan create -pr:b default -pr:h default . celix/2.2.3@zhengpeng/testing -b missing.

Resolving dependency conflicts in downstream is generally correct. Here the mentioned conflict should not appear at the first place.

@PengZheng
Copy link
Contributor Author

IDE integration is essential for Conan-based CBD. Adding a requirement, then happily including header and invoking APIs are huge productivity improvement. Here I summarized my own Clion setup to facilitate such use cases. Hope that will help other developers. @pnoltes

  1. Setup all dependencies using Conan:
(conan) peng@hackerlife:~/Downloads/git/mycelix$ conan install . celix/2.3.0@zhengpeng/testing -pr:b clang_debug -pr:h clang_debug -if cmake-build-debugwithtestsclang/ -o celix:enable_testing=True -o celix:enable_address_sanitizer=True -o celix:build_all=True -b missing

Here -pr:b and -pr:h are essential for tool_requires in conanfile.py to work.

  1. Translate all conan options into CMake Cache without actually building:
conan build . -bf cmake-build-debugwithtestsclang --configure
  1. Setup Clion CMake profiles properly, special care should be taken to make build directory consistent with that in previous steps, i.e. cmake-build-debugwithtestsclang.
    CMakeSettings

  2. Setup Run/Debug Configurations of the target you want to debug. Special care should be taken to setup environment variables properly, otherwise shared libraries in Conan cache can not be found. We choose to use activate_run.sh under the build direcotry.
    debug

  3. Happy debugging!
    debugging

@PengZheng PengZheng merged commit 4e70917 into master Apr 4, 2022
@PengZheng PengZheng deleted the feature/conan_support branch April 4, 2022 08:05
@pnoltes
Copy link
Contributor

pnoltes commented Apr 4, 2022

Thanks for the extra info. Now I also got it running in CLion 😄.

I am not very happy that this is needed for all the run/debug configurations, but (again) this is something we can try to tackle later.

I will try to also include the conan information in the coming updated documentation pr.

@PengZheng
Copy link
Contributor Author

I am not very happy that this is needed for all the run/debug configurations, but (again) this is something we can try to tackle later.

Oops!

We should setup the All CTest target rather than any specific test target. Run All CTest target once, then we can right click any specific test in Run(alt+4) window to debug it, an Run/Debug Configuration will be created automatically for the test we select.
AllTest

HTH @pnoltes

@PengZheng
Copy link
Contributor Author

I'm writing to notify that there is a recently introduced regression of CLion 2022.1, which affects environment variable setting:
https://youtrack.jetbrains.com/issue/CPP-29003

It will be fixed for the next release, and a workaround is provided by the above link. @pnoltes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants