Add cpack functionality to CMake build helpers

[KerstinKeller]

In our local workflow, we want to use conan for managing (build) dependencies, but (for now) we still want to create also deployment artefacts (e.g. debian packages or Windows installers).
(Similar to this ticket #3541)

So I am thinking of adding a deploy() functions to my recipes, which will call CPack.
For this it would be very helpful, if the CMake build helper also gained a package or cpack method. (This is different to cmake.install() as it will install to a specified folder.)
It would be very similar to the test/install methods, but instead of building the RUN_TESTS/INSTALL targets, it needs to build the PACKAGE target.
Also I am not quite sure if this method needs any arguments and which those would be.
(E.g. on Linux I often explicitly call cpack -G DEB.
Would you consider adding such functionality?

However, I am not quite sure if the deploy() method would be the correct method to call this cmake.package() step, because according to the documentation deploy() gets executed when a dependent package calls conan install on that package.
I guess it would be "ortogonal" to the package() function, like a non_conan_package(). That wouldn't be executed on conan create, but rather for the local development workflow.

• I've read the CONTRIBUTING guide.
• I've specified the Conan version, operating system version and any tool that can be relevant.
• I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

[jgsogo]

Hi! Up to now, we haven't developed anything specific to packaging, we usually recommend to use the deploy generator (https://docs.conan.io/en/latest/reference/generators/deploy.html) and an external script to make the final bundle/package/installer/...

Nevertheless, that approach works only with the final artifacts, not with the CMake project, so this use-case is quite different, here we need to run it during creation of a package (stages build or package) because we need CMake to call it.

Right now conan build command has arguments --configure, --build, --install, --test that populate variables should_configure, should_build, should_install and should_test. The natural way to implement your request would be to add a should_package variable tied to a --package argument (better naming needed).... although it is somehow confusing to package during build stage.

Maybe it is more natural to use the package() function and the local command package, a flag argument can be added to it and a new variable will be available in the function. During this command, the CMake project should be available in the build directory, so we can build the PACKAGE target.

These are just ideas, not a solution.

ping @SSE4, he will find this issue interesting as he's been gathering information about different packaging approaches.

[KerstinKeller]

Yes, I agree that it is confusing to to call package during the build stage. Same as to call the install method during build stage, just as most my recipes do 🤔

The CMake build helper can be made available in all functions, and then of course we can call the cmake.package() in the conanfile.py package() method.
However, everything happening in the package() method so far, will be part of the consumable conan package. But in this case, the output is more additional to current conan package.
It should not be part of the package, but more like an extra, additional artefact.

However, if it was bundled with the package, it could be later used in a deploy step, e.g. instead of collecting .so files together for a runtime, you could collect all .deb packages from the dependencies and then have a bundle of .deb packages to install for an actual deployment.
It's like an artefact which is not necessary for the actual building of additonal packages that rely on that package, but only for creating a final deployment.
But just thinking out loud here....

The deployment on Linux is really still a pain point for our Conan workflow, because so far our team has not come to a good conclusion how to handle this. The last thing we want is to have a giant package with all .so files included, because it feels very unlike what you would do on Linux.
Distributing a bundle of .deb package files sounds a lot better, because you can install / uninstall them one by one. If you deploy two tools which depend on the same shared library we create two bundels (tool1.deb, libA.deb), (tool2.deb, libA.deb), the shared library will be in it's own debian package and there will be no conflicts installing them to the same system path prefix, because libA.deb will get installed only once.

It does have some overhead for creating the recipes, and we will have to make sure the libraries can be packaged with CPack. If not we might be able to fall back to using checkinstall for non-self-maintained libraries.
So yeah, I guess if you could make the package method available on the cmake helpers, to be called in the package() function of the conanfile, it could be very helpful!

On Windows, we will just copy together all the dlls, and be done with it 😄

[uilianries]

Hi @KerstinKeller !

You could use Conan hooks to run cpack after packaging your artifacts. Many time ago, when hooks was not available, I implemented the follow routine:

def package(self):
    ...

    def package_install(self):
        if self.settings.os == "FreeBSD":
            self.run("cpack -G TXZ")
            self.copy("%s-%s.tar.xz" % (self.name, self.version), dst="package")
        else:
            package_type = 'RPM' if isfile('/etc/redhat-release') else 'DEB'
            self.run('cpack -G %s ..' % package_type)
            self.copy("%s-%s.%s" % (self.name, self.version, package_type.lower()), dst="package")
    
    package_install()

But now you try something more elaborated and safe:

def post_package(output, conanfile, conanfile_path, **kwargs):
    if self.settings.os == "FreeBSD":
        conanfile.run("cpack -G TXZ")
        conanfile.copy("%s-%s.tar.xz" % (conanfile.name, conanfile.version), dst="package")
    else:
        package_type = 'RPM' if os.path.isfile('/etc/redhat-release') else 'DEB'
        conanfile.run('cpack -G %s ..' % package_type)
        conanfile.copy("%s-%s.%s" % (conanfile.name, conanfile.version, package_type.lower()), dst="package")

UPDATE:

On Windows you can use Wix to generate .msi files, much better than NSIS.

Reference: https://docs.conan.io/en/latest/reference/hooks.html

Regards!

[KerstinKeller]

Hi @uilianries,

thanks for the suggestion. But I am not quite sure I fully understand.
In my understanding, we have hooks for performing common tasks, applicable to all packages, non-intrusively.
So we could check if certain naming conventions are followed for package name and version number.
We could check also if a header only package contained only header files, or that all packages contain some kind of License file.

But actually changing still the package (e.g. adding the .deb) file to the package, I find a bit strange. But so far, I have not used any hooks, so not sure here. What kind of things are you generally performing with the Hooks?

I have no problem implementing a hook the way you showed, but I wonder if CPack can always be called generically, without extra arguments.

Also conceptionally, I wonder what to do with the debs. Are they really part of the package, or are they an extra asset connected with the package. I might want to upload them to Artifactory separately, but I will definately want to collect them when creating a bundle for my application.
And here I have not yet found a proper way to model it with Conan.

And also thanks for mentioning Wix, we've already been using it on Windows. I agree it's better than NSIS, but there is way too little documentation out there for CPack integration. Expecially when performing tasks like setting Enviroment variables with the installer, adding custom icons etc. it's all trial and error 😄

[uilianries]

Hi @KerstinKeller !

But actually changing still the package (e.g. adding the .deb) file to the package, I find a bit strange. But so far, I have not used any hooks, so not sure here. What kind of things are you generally performing with the Hooks

I agree, packaging .deb or any other installer file does not make any sense. But you can use the hook to upload to another server, like Bintray for general distribution. They offer free public repositories for organizations. About hooks, you can use to execute extra tasks which are not part of your recipe, like run a linter to check the quality of your recipe, sending an email/notification after uploading a new package version. We have a bunch of official hooks here.

I have no problem implementing a hook the way you showed, but I wonder if CPack can always be called generically, without extra arguments.

I see, CPack offer a big amount of variables which can be configure in your cmake file. In my case, ALL variables were configured in my file, so I didn't need any extra definition when calling cpack. For example:

# CMakeLists.txt
include(InstallRequiredSystemLibraries)
set(CPACK_PACKAGE_DESCRIPTION "My super project")
set(CPACK_PACKAGE_VERSION_MAJOR "${VERSION_MAJOR}")
set(CPACK_PACKAGE_VERSION_MINOR "${VERSION_MINOR}")
set(CPACK_PACKAGE_VERSION_PATCH "${VERSION_PATCH}")
set(CPACK_PACKAGE_VENDOR "ACME")
set(CPACK_PACKAGE_CONTACT "software@acme.com")
set(CPACK_PACKAGE_FILE_NAME "${CMAKE_PROJECT_NAME}-${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}")
include (CPack)

You can find all supported variables here.

Also conceptionally, I wonder what to do with the debs. Are they really part of the package, or are they an extra asset connected with the package. I might want to upload them to Artifactory separately, but I will definately want to collect them when creating a bundle for my application.
And here I have not yet found a proper way to model it with Conan.

.deb or any other installer is separated from the package, because you would exporting a package in other package. So thinking again, instead of copy, you could use Bintray API to upload your installer and discard it from your local cache.

And also thanks for mentioning Wix, we've already been using it on Windows. I agree it's better than NSIS, but there is way too little documentation out there for CPack integration. Expecially when performing tasks like setting Enviroment variables with the installer, adding custom icons etc. it's all trial and error

In fact, the documentation is limited, but CMake offers a list of supported variables for CPackWIX. I remember that I spent few days configuring my first project with Wix, because I had some trouble installing Candle + Wix. The advantage is the .msi product, it's a native Windows installer, so any user can use the command line to install, much easier than a setup.exe with dialog boxes.

[SSE4]

I've been talking with debian and fedora package maintainers, and they strongly discourage the usage of CPack for .deb or .rpm generation. the reasons are:

• it doesn't create .deb or .rpm package in archive format
• it doesn't add metadata to the created packages
• it doesn't write dependencies from libraries or other packages
• it doesn't export provides
• it doesn't handle mime-types
• it doesn't add mandatory scriptlets
• it doesn't follow the official guidelines of distributions

source (in Russian)

[uilianries]

Great! Thanks @SSE4 for bringing this valuable info!

[KerstinKeller]

Hi @SSE4,

thanks for the info. I don't know if this thread is the correct point for discussion of this topic, but let me add a few points.

So first I think you need to distinguish: do you want to create Debian packages to make contribution to official distributions, or do you want to create packages for an easy / safe way to provide your prebuild software for consumption on multiple machines inside your corporate network.

Also, I know too little about traditional Debian package generation, to really argue the points. But I know, that with CMake, you can do a lot of things among which the above post claims that they aren't possible.

• it doesn't create .deb or .rpm package in archive format I don't know what it meant by that point. I can double click the generated .deb file and it opens with the archive extractor....
• it doesn't add metadata to the created packages The created deb contains a control.tar.gz folder, which has all the control files, such as control, conffiles, ... Do official .deb packages contain more data?
• it doesn't write dependencies from libraries or other packages By default it doesn't, but you can set the CPACK_DEBIAN_PACKAGE_SHLIBDEPS variable to ON. Then they are automatically added to the control file. (e.g. it automatically adds libhdf5-10, libqt5core5a (>= 5.5.0) ...).
  Also you can say that the shlibs file may be automatically created by setting CPACK_DEBIAN_PACKAGE_GENERATE_SHLIBS to ON.
  Big drawback though, when building with Conan though, is you build against conan packages, so in that case you have to specify your dependencies manually by setting the CPACK_DEBIAN_PACKAGE_DEPENDS variable.
• it doesn't export provides According to the documentation, you can add provides with the following variable CPACK_DEBIAN_PACKAGE_PROVIDES. Haven't tried that one myself though.
• it doesn't handle mime-types No clue about that one. But I guess that you could handle them too. CMake let's you install almost anything you want, so you could very well also create / install a <package>.sharedmimeinfo file, though manually.
• it doesn't add mandatory scriptlets No clue about that one.
• it doesn't follow the official guidelines of distributions Hm, no clue either.

I think the biggest drawback with CPack Debian (or with CPack in general) is the poor documentation and the lack of examples.
If you have a good template for your projects, you can create decent, usable packages with very little manual overhead, and basically create CPack packages "for free".

Unfortunately, I am not able to read / understand Russian in order to understand the original conversation 😞

[memsharded]

With the new extensions points in Conan 2.0 (custom commands, deployers), this will not be implemented built-in in Conan. If any effort is done, I'd recommend doing it in the extensions repo: https://github.com/conan-io/conan-extensions.

Closing this ticket, thanks!