MINIFICPP-2346-P2: Added Support to Create MiNiFi Conan2 Package, Easier Integration into C++ RT Infra #1813
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MINIFICPP-1872 Unify test utilities - Remove OpenSSL and Curl options and make them part of the libminifi core - Remove http extension and make it part of libminifi core and InvokeHTTP processor part of standard-processors - Move HTTP tests as part of libminifi integration tests - Extract test library in libminifi, with separate integration and unit test library - Make test utilities part of the test library - Move all tests to their proper unittest or integrationtest folder in libminifi - Convert integration tests to use the catch2 framework and remove executable parametarization
…i binary via Conan v2
Verified I can also build MiNiFi using standalone CMake still
Addressed martinzink's feedback adding GetLibCURL, GetOpenSSL and GetZLIB cmake files that use prebuilt conan package when MINIFI_USE_CONAN_PACKAGER is set, otherwise it uses bundled cmake files from standalone cmake approach. After updating the files, I verified I could build MiNIFi using conan build and then verified I could build MiNiFi using standalone CMake approach. Furthermore, I also verified I could run minifi binary
…ONAN_PACKAGER tc variables can be used to change minifi options
After addressing some feedback and working with Marton on some build issues he was facing with conan on his side, I realized that conan's OpenSSL, ZLib and LibCURL were being ignored because I didn't update conanfile.py file's USE_CONAN_PACKAGER to be MINIFI_USE_CONAN_PACKAGER to match with MiNiFiOptions.cmake one. Once I made that I update then when building MiNiFi C++, it shows in the build output for OpenSSL, 'Using Conan Packager to manage installing prebuilt OpenSSL external lib' and similar messages for zlib and libcurl. Thus, OpenSSL, ZLib and LibCURL arent build from source since we use the conan prebuilt ones. Also I verified that I can build MiNiFi C++ using default conan v2 profile using the similar steps Marton tried. The main reason I suggested we use a particular conan profile in MiNiFi C++ is so each environment where MiNiFi C++ is built has a consistent conan profile since each environment default generated conan profile could be unique to that environment, which could cause issues in downloading prebuilt conan packages.
…URCE vars
Was able to build MiNiFi C++ with the ENABLE_{External_Lib} variables set to OFF initially.
However, after a suggestion from Marton to leave the ENABLE external lib variables set to
their default values from MiNiFiOptions.cmake, I removed the tc.variables['ENABLE_{...}']
that overrode MiNiFiOptions.cmake ones and now am trying to build MiNiFi C++ using conan
where we use conan's OpenSSL, ZLib and LibCURL prebuilt conan packages alternative to building
them from source. I am debugging issues that come up from building MiNiFi using conan where
partly we use conan's prebuilt conan packages and then the remaining external libs are the ones
built from source. For instance, one finding related to needing to use the predefined conan profile
located in etc/conan/profiles/release-linux because of the gnu20 was required to build rocksdb
successfully. I found that for my environment, conan's generated default profile used gnu17,
which caused rocksdb to fail building, but setting conan to use gnu20 allowed it to succeed in
building rocksdb.
After I added new MiNiFi options for MINIFI_LIBCURL_SOURCE, MINIFI_OPENSSL_SOURCE and MINIFI_ZLIB_SOURCE all set to CONAN, I tried building MiNiFi with all the default ENABLED options, but there were two I needed to switch to OFF. I had to switch ENABLE_LIBARCHIVE and ENABLE_AWS to OFF because they failed to build. I do want to note that on my much larger PR where I built majority of MiNiFi using conan to install most of the external libs, I was able to build MiNiFi with openssl, libcurl, zlib and libarchive with no issues. I was also able to keep AWS enabled. I can do a follow PR where I bring back ENABLE_LIBARCHIVE and ENABLE_AWS to where we can set them to ON and be able to successfully build MiNiFi using conan
…to Other C++ Infrastructure Currently, MiNiFi C++ mainly acts as a standalone project, but I think it would also be useful to be able to create MiNiFi CPP as a conan package for easier integration of MiNiFi CPP into other C++ real-time infrastructure. Once MiNiFi C++ is integrated as a library into other C++ infrastructure through conan package, we can create MiNiFi edge data pipelines programmatically in C++ as an alternative to creating them through yaml files. There are already a lot of examples through GTests of MiNiFi data pipelines being created programmatically, I think it would be useful for companies to explore bringing the best of MiNiFi and NiFi data pipeline programming into their infrastructure. I verified I can create the minifi-cpp/0.99.0@minifi/develop conan package. Next I will explore running a test package example by programmatically creating a MiNiFi CPP data pipeline.
james94
changed the title
After I create a MiNiFi C++ conan package, conan then compiles the C++ code in the test_package, which contains 2 C++ test example programs from MiNiFi. The first program is AbstractProcessorTest from libminifi and the second program is MinifiGenerateFlowFileTest.cpp. conan will successfully build the test_package for MiNiFi C++ conan package and link minifi-cpp lbrary aka the conan package to be used in running these 2 C++ test programs. AbstractProcessorTest succeeds and I believe MinifiGenerateFlowTest fails on some tests because we are missing some resource dependencies that are typically available in nifi-minifi-cpp repo, but currently not available in the minifi-cpp conan package. I just needed to verify I could use the minifi-cpp conan package to run a few minifi-cpp test programs, especially the test program that tests the GenerateFlowFile C++ processor within a minifi-cpp edge data pipeline that was programmatically created. This update should make it easier now for other companies to bring MiNiFi CPP into their C++ infrastructure as a library as an alternative to bringing MiNiFi in as the standalone program
By having the .ipp files included in the minifi-cpp conan package, now I can build expression language tests and use expression language features from minifi-cpp conan package from within another C++ software infrastructure.
|
Hi @james94, sorry this PR was closed automatically when I deleted the apache:MINIFICPP-2251 branch (as it was merged a while back) it was based on. Please rebase it to the main branch and reopen the PR. |
james94
commented
Jul 7, 2024
| cmake.configure() | ||
| cmake.build() | ||
|
|
||
| def package(self): |
There was a problem hiding this comment.
Here is conan's way of packaging assets after building MiNiFi C++ where we can tell conan to collect the headers, library (static, dynamic, etc), app binaries, etc @szaszm @lordgamez
In CMake standalone, we would build MiNiFi C++ and then run make package to create MiNiFi binary tarball.
However, by adding this ConanFile package method, we are able to run conan create minifi-cpp ..., which can then be uploaded to a conan repository like conancenter, jfrog, github, etc. Whether we now use the local minifi-cpp conan package or remote one, now we can easily integrate minifi-cpp into other C++ software infrastructures through their conanfile.py.
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@szaszm @martinzink Great news with my recent update, it still is below 1000 updates and believe still eligible for review. This PR is meant to go on top of james94:MINIFICPP-2346-P1.
Here is the command to create the minifi conan package:
UPDATE: (June 14, 2024): Verified Can Create MiNiFi conan package & run 2 test programs AbstractProcessorTest and MinifiGenerateFlowFileTest, which are both tests from MiNiFi repo now just built and run from conan test_package.
This PR goes right on top of my smaller "PR-1793: MINIFICPP-2346-P1: Integrated Conan2 for OpenSSL, CURL, ZLIB & Build MiNiFi", it just adds the ability for us to use Conan to create for instance a minifi-cpp/0.99.0@minifi/develop conan package for easier integration of MiNiFi C++ into other C++ real-time infrastructure.
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:
Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?
For code changes:
For documentation related changes:
Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.