Remaken is a meta dependencies management tool.
Remaken supports various C++ packaging systems using a standardized dependency file format (named packagedependencies), that follows a modular syntax.
Remaken also provide its own C++ packaging structure, based on pkg-config description files.
Remaken can be seen as the "one tool to rule them all" for C++ packagers.
The need to use a tool such as Remaken comes from :
- diversity of C++ package managers
- need to speed up development bootstrap
- need to shorten delays
Indeed, whatever the dependency your project needs, there is a chance the dependency already exists in binary format on one of the existing C++ packaging systems.
If it doesn't already exist in an existing C++ packaging systems, you can build your own binary version and share it as a remaken dependency across your team or organization, ensuring build options are the same.
It also avoids other developers to build locally the same dependency.
To setup a native C/C++ project that uses thirdparty dependencies, a developer must for each dependency (whatever the build system is in either make, cmake, conan, vcpkg, MSVC …):
- build each dependency with homogeneous build rules and flags (for instance c++ std version, for each target [debug, release | os | cpu | shared, static])
- install each dependency in an accessible path in the system (and eventually, pollute the development environment system for instance using
make install, or set a different sysroot - make install by default will need sudo access for unix(e)s.) - add include paths, libraries paths and libraries link flags for each dependency and sub dependency in its development project
- For each new development, even based on same dependencies : reproduce every dependency build step 1 to 3 (also true for conan when the binary hosted version doesn’t fit with your options)
- Running a final application : each dependency must either be copied in the same folder than the application or paths must be set to each shared dependency folder.
- Last but not least, to package a final application with its dependencies means to bundle manually every shared dependency with the application before creating the final bundle/setup for the application
NOTE: for most build systems (make, cmake, Microsoft Visual Studio builds ...) the target can be overwritten across versions. The solution would be to manually add custom target path to installation post-build steps - for each project.
Shared library or application project heterogeneity across a team can lead to integration issues as the build rules can slightly differ on each developer station.
Remaken :
- provide the same set of build rules across packaging systems (certified for packaging systems that perform a local build such as conan, vcpkg, brew)
- install dependencies from any packaging system (including remaken own packaging and flag finding system)
- there is no need to call manually each packaging system, to write a script … all is done at once from the packagedependencies description
- build flags are populated either from remaken configure step or from builddefs-qmake rules (or other rules format will come in the future such as cmake, bazel ...)
- run any application with appropriate environment deduced from described dependencies and/or from xpcf configuration file
- automate dependencies bundling within an application from dependencies description (either copying external dependencies in bundle destination folder or creating a shell script for native package managers such as apt, yum ...)
- bundle xpcf applications from xpcf configuration file
- integrate conan dependencies easily without writing a conanfile.py
- provide a normalized package installation structure for remaken dependencies. cf Package tree
- allow to manage several dependencies version at the same time (each installation is based and searched with the package version number)
- binaries installation don't occur in system path. It avoids the pollution of the environment. It also avoids the need for sudo access.
- provide vcpkg installation and bootstrap
- provide builddefs-qmake rules installation and update
First install brew on your system:
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
Then enter the following commands from a terminal:
brew tap b-com/sft
brew install remaken
wget https://github.com/b-com-software-basis/remaken/releases/download/install/install_remaken.sh
./install_remaken.sh
Download and install the latest setup file version.
- conan
- vcpkg
- brew (mac OS X and linux)
- Remaken packaging structure (pkg-config based)
- apt-get (debian and derivatives)
- pacman (archlinux)
- yum (redhat, fedora, centos)
- zypper (openSUSE)
- pkg (freebsd)
- pkgutil (solaris)
- chocolatey (Windows)
- scoop (Windows)
- [others to come]
remaken init: creates the default .remaken folder.
It also initializes .remaken/rules/qmake with the builddefs-qmake release associated with the current remaken version. To get another version, use remaken init --tag x.y.z.
You can also retrieve the latest qmake rules (latest rules reflects the current develop branch status) with remaken init --tag latest.
All qmake rules are retrieved from builddefs-qmake.
The remaken init command also supports the --force (alias -f) to force reinstalling qmake rules and/or --override (alias -e) to override existing files.
Note: remaken init need to be called with elevation rights (admin/sudo mode)
remaken init vcpkg: initializes vcpkg (clone the vcpkg repository and bootstrap vcpkg)remaken init vcpkg [--tag tag]: initializes vcpkg (clone the vcpkg repository for tag [tag] and bootstrap vcpkg)
For now, only QtCreator wizards are provided.
To install the remaken, xpcf projects and classes wizards, use:
remaken init -wremaken init wizards
To install the artefact packager scripts for current OS in .remaken/scripts :
remaken init artefactpkg: initializes artefact packager script (download script on https://github.com/b-com-software-basis/remaken/releases/tag/artifactpackager)remaken init artefactpkg [--tag tag]: initializes artefact packager script (download script on https://github.com/b-com-software-basis/remaken/releases/tag/artifactpackager for tag [tag])
The remaken packages folder is the location where packages are installed, searched from ...
This folder defaults to $(HOME)/.remaken/packages.
It can be changed in various ways:
- define the
REMAKEN_PKG_ROOTenvironment variable to another location. - create a
.packagespathfile in remaken root folder$(HOME)/.remaken. In this file, provide the path to your packages folder in the first line. - provide the package path with the
--remaken-rootparameter from remaken command line invocation. - put the
remaken-rootparameter and its value in the remakenconfigfile in located in$(HOME)/.remaken.
Remaken has profile support to set several options once for all for every projects (for instance, the os, the architecture, the C++ standard ...).
It also supports the creation of named profiles - it can be used for cross-platform builds (to target android systems for instance).
To initialize the default profile, use:
remaken profile init
You can also specify any value you want to set in the profile:
remaken profile init --cpp-std 17remaken profile init --apiKey XXX# artifactory apiKey can be set in the profile
To create a named profile, use:
remaken profile init [profile_name or path]
To use a named profile for any command, run:
remaken --profile [profile_name or path] ...
To display the default profile configuration:
remaken profile display
To display a named profile configuration:
remaken --profile [profile_name or path] profile display
To display options from the profile file and from default remaken values:
remaken profile display -w
remaken search [--restrict packaging_system_name] package_name [package_version]
packaging_system_name is the name of the packaging system to search for this dependency. Its value depends on the operating system where remaken is used.
On mac OS X and linux, its value is one of [brew, conan, system or vcpkg].
On windows, its value is one of [choco, conan, scoop, system or vcpkg].
To get the list of installed remotes for every packaging system installed on the environment run:
remaken remote list
To get the list of remotes declared in a packagedependencies.txt, run (using --recurse will list recursively every remote declared in every packagedependencies file in the dependency tree):
remaken remote listfile [--recurse] [path_to_remaken_dependencies_description_file.txt]
To add the remotes declared in a packagedependencies.txt, run (using --recurse will add recursively every remote declared in every packagedependencies file in the dependency tree):
remaken remote add [--recurse] [path_to_remaken_dependencies_description_file.txt]
To get a sample how to declare an additional remote/bucket/ppa ... see the repository_url section in chapter Dependency file syntax
-
remaken install [--conan_profile conan_profile_name] [-r path_to_remaken_root] -i [-o linux] -t github [-l nexus -u http://url_to_root_nexus_repo] [--cpp-std 17] [-c debug|release] [-m static|shared] [--project_mode,-p] [path_to_remaken_dependencies_description_file.txt] [--condition name=value]* [--conan-build dependency]* -
remaken configure [--conan_profile conan_profile_name] [-r path_to_remaken_root] -i [-o linux] -t github [-l nexus -u http://url_to_root_nexus_repo] [--cpp-std 17] [-c debug|release] [-m static|shared] [--project_mode,-p] [path_to_remaken_dependencies_description_file.txt] [--condition name=value]*
Notes:
- remaken_root defaults to
$(HOME)/.remaken/packagesor ifREMAKEN_PKG_ROOTenvironment variable is defined, it defaults to${REMAKEN_PKG_ROOT}.
[-r path_to_remaken_root]allows to specify a specific Remaken packages root directory. path_to_remaken_dependencies_description_filedefaults to current folderpackagedependencies.txtfile[--project_mode,-p]: enable project mode to generate project build files from packaging tools (conanbuildinfo.xxx, conanfile.txt ...).
Project mode is enabled automatically when the folder containing the packagedependencies file also contains a QT project file[--conan_profile conan_profile_name]allows to specify a specific conan profile
When--conan_profileis not specified:- for native compilation: the
defaultconan profile is used - for cross compilation: remaken expects a
[os]-[build-toolchain]-[architecture]profile to be available - for instance for android build, expects aandroid-clang-armv8profile
- for native compilation: the
-ifor ignore cache, then dependencies update is forced[-l nexus -u http://url_to_root_nexus_repo]allow s to specify alternate remote type, and alternate remote url
[--invert-remote-order]allows to invert alernate remote type and url with default remote type and usrl used in packagedependencies file[--conan_build dependency]is a repeatable option, allows to specify to force rebuild of a conan dependency. Ex :--conan-build boost.[--condition name=value]is a repeatable option, allows to force a condition without application prompt (useful in CI). Ex :--condition USE_GRPC=true.
By default, remaken install and remaken configure prompt users for enabled or disabled configure conditions.
Note:
A condition is defined just after library name with library name%[condition] in packagedependencies.txt file
a configure_conditions.pri file can be added into the project root folder to avoid user prompt :
DEFINES+=CONDITION1
DEFINES-=CONDITION2
of
After remaken install or remaken configure, a configure_conditions.pri file is generated in .build-rules/[os]-[build-toolchain]-[architecture] folder with specified (by file or by user_prompt) enabled conditions.
The generated file is used when run remaken install or remaken configure or qmake (on current project) for enable/disable dependencies automatically.
Then as only dependencies used are in the generated .build-rules/[os]-[build-toolchain]-[architecture]/packagedependencies.txt, only enabled condtions are generated in this file.
Note: [--condition name=value] allows to force a condition without application prompt (useful in CI). Ex : --condition USE_GRPC=true.
Also generates build flags in remaken build rules folder (.build-rules/[os]-[build-toolchain]-[architecture]) :
- a master file
dependenciesBuildInfo.priincludesconanbuildinfo.pri: conan dependencies flagsremakenbuildinfo.pri: remaken dependencies flags
This file can be used automatically in qmake projects by using use_remaken_parser value in DEPENDENCIESCONFIG, CONFIG or REMAKENCONFIG value.
It disables package dependencies parsing made by builddefs-qmake rules.
remaken bundle [--recurse] [-d path_to_bundle_destination] [--cpp-std 11|14|17|20] [-c debug|release] [path_to_remaken_dependencies_description_file.txt]
Note: remaken_dependencies_description_file defaults to current folder packagedependencies.txtfile.
remaken bundleXpcf -d path_to_root_destination_folder -s relative_install_path_to_modules_folder --cpp-std 17 -c debug xpcfApplication.xml
remaken parse [path_to_remaken_dependencies_description_file.txt]
remaken clean
remaken info [path_to_remaken_dependencies_description_file.txt]: displays the recursive dependency tree from the file.
remaken info pkg_systemfile -d path_to_write_pkgfile [path_to_remaken_dependencies_description_file.txt]: generates pkg system files from packagedependencies files into specified folder - currently Only conan is managed.
remaken info [path_to_remaken_dependencies_description_file.txt] [-p [-m static|shared]]: displays include, lib, and bin paths of all dependencies. can take an optionnal mode option (static or release) for defined default use mode.
The list command allows to :
remaken list: list all remaken installed packagesremaken list [package name]: list all installed versions of a remaken packageremaken list --regex [package name regular expression]: list all installed versions of remaken packages whose names match a regexremaken list [package name] [package version]: list all files for a specific version of a remaken package
remaken list also has a --treeflag. When this flag is set, remaken list displays packages informations and their dependencies tree.
remaken can be used to ease application run by gathering all shared libraries paths and exposing the paths in the appropriate environment variable (LD_LIBRARY_PATH for unixes, DYLD_LIBRARY_PATH for mac and PATH for windows)
remaken run [-c debug|release] run --env [--ref remaken package reference] [--deps path_to_remaken_dependencies_description_file.txt] [--xpcf path_to_xpcf_configuration_file.xml] [--app path_to_executable] [--name ]-- [executable arguments list]
remaken also allow to run an application installed in remaken packages structure from its package reference (i.e [package name:package version]).
remaken run [-c debug|release] --ref [package name:package version]
In this configuration, dynamic dependencies are searched in the package dependencies file and added to the platform library search path.
If an xpcf xml file is installed in the package folder or next to the application binary, the xpcf xml file is parsed to add the modules and their dependencies to the platform library search path.
For instance: remaken run -c debug --ref testxpcf:2.5.1
The --ref option can be used with other options, especially to overwrite the application name if it differs from the package name, or to retrieve the environment set using --env.
--app represents the complete application filepath including the filename.
Note : options in [executable arguments list] starting with a dash (-) must be surrounded with quotes and prefixed with \ for instance "\-f" to forward -f option to the application (this is due to CLI11 interpretation of options).
For each project, a packagedependencies.txt file can be created in the root project folder.
For dependencies specific to a particular os, a packagedependencies-[os].txt can also be created in the root project folder (os is a value in {android, linux, mac, win } ).
An extra-packages.txt file can also be created along the packagedependencies.txt file.
The extra-packages.txt file can be used to add dependencies to install and bundle steps. Dependencies listed in the extra-packages.txt file will not be used by configure steps, or by the project build rules.
The extra-packages.txt purpose is to:
- install and bundle sub-dependencies needed by a project direct dependency when the direct dependency doesn't install its sub-dependency. (for instance, the conan opencv recipe doesn't install gtk+ on linux, but gtk+ is needed when a project uses opencv::imgui, hence the opencv conan recipe is "incomplete" as the imgui functionality is disabled when the gtk+ package is missing)
- install non-dependency packages : for instance install data needed by an application, configuration files ...
For packages specific to a particular os, an extra-packages-[os].txt can also be created in the root project folder (os is a value in {android, linux, mac, win } ).
An packagignoreinstall.txt file can also be created along the packagedependencies.txt file.
The packagignoreinstall.txt can be used to ignore dependencies to install at bundle steps.
Dependencies listed in the packagignoreinstall.txt file will not be used by configure steps, or by the project build rules.
Each framework_name are ignored (as defined in packagedependencies.txt file) with the pattern
framework1 framework2
or
framework1
framework2
The project build rules (builddefs-qmake for instance) will generate a packagedependencies.txt containing the build informations and will gather the dependencies in the original packagedependencies.txt and packagedependencies-[os].txt for the target os the build is run for.
Each line follows the pattern :
framework#channel | version | library name%[condition] | identifier@repository_type | repository_url | link_mode | options
framework[#channel] |
version |
library name%[condition] |
repository_type |
repository_url |
link_mode |
options |
|---|---|---|---|---|---|---|
| the framework name. Optionnally the channel to get the package from | the version number | the library name | a value in: [ artifactory, nexus, github or http (http or https public repositories), vcpkg, conan, system, path : local or network filesystem root path hosting the dependencies ] | --- | optional value in : [ static, shared, default (inherits the project's link mode), na (not applicable) ] | --- |
Note:
To comment a line (and thus ignore a dependency) start the line with // (spaces and tabs before the // are ignored).
NOT IMPLEMENTED : For artifactory, nexus and github repositories, channel is a named scope describing a common combination of compile options from the remaken packaging manifests. The combination of values become a named scope. (TODO : manage named scopes) It is not used for other kind of repos.
framework#channel |
|---|
| when channel is not specified, it defaults to stable for conan dependencies |
repository_type |
|---|
| When repository_type is not specified : - it defaults to artifactory when identifier is either bcomBuild or thirdParties (and in this case, the identifier is also the destination subfolder where the dependencies are installed) - it defaults to system when identifier is one of yum, apt, pkgtool, pkgutil, brew, macports, pacman, choco, zypperFor other repository types (github, vcpkg, conan, system) When the identifier matches the repository type,the repository type reflects the identifier value - i.e. identifier = conan means repository_type is set to conan. When identifier is not specified :- @repository_type is mandatory |
repository_url |
|---|
| repository url can be: - the repository url for github, artifactory or nexus repositories For brew, scoop, apt and conan, the repository will be added before the dependency installation. For those systems, the url format is: - for brew taps it will be either user/repo or user/repo#repository_url - for scoop buckets it will be either repo_identifier or repo_identifier#repository_url - for conan it will be repo_identifier#repository_url[#position] - for apt it will be the ppa url |
options |
|---|
| options are directly forwarded to the underlying repository tool. Note : to provide specific options to dedicated system packaging tools, use one line for each specific tool describing the dependency. (once installed, system dependencies should not need specific options declarations during dependencies' parsing at project build stage. Hence the need for the below sample should be close to 0, except for packaging tools that build package upon install such as brew and macports and where build options can be provided). |
samples |
|---|
| eigen | 3.3.5 | eigen | system | https://github.com/SolarFramework/binaries/releases/download |
| eigen | 3.3.5 | eigen |
| eigen | 3.3.5 | eigen | pkgtool@system | https://github.com/SolarFramework/binaries/releases/download | default | --S#--noconfirm |
| opencv | 3.4.3 | opencv | thirdParties@github | https://github.com/SolarFramework/binaries/releases/download |
| xpcf | 2.1.0 | xpcf | bcomBuild@github | https://github.com/SolarFramework/binaries/releases/download | static |
| spdlog | 0.14.0 | spdlog | thirdParties@github | https://github.com/SolarFramework/binaries/releases/download |
| eigen | 3.3.5 | eigen | system | |
| fbow | 0.0.1 | fbow | vcpkg | |
| boost#stable | 1.70.0 | boost | conan | conan-center |
| spdlog | 0.14.0 | spdlog | bincrafters@conan | conan-center | na |
| freeglut#testing | 3.0.0 | freeglut | user@conan | https://github.com/SolarFramework/binaries/releases/download |
github, artifactory, nexus and path dependencies are installed using remaken packaging format through an url or filesystem repository.
system dependencies are installed using operating system dependent package manager (apt for linux debian and derivatives, brew for Mac OS X, chocolatey for windows...)
conan dependencies are installed using packaging format with conan package manager. conan url must match a declared remote in conan (remotes added with conan remote add).
vcpkg dependencies are installed using vcpkg packaging format with vcpkg package manager
WARNING : using system without any OS option implies the current system the tool is run on. Moreover, some OSes don't have a package manager, hence don't rely on system for android cross-compilation for instance.
PackageName
|- PackageVersion (X.X.X)
|-- interfaces
|--- [here your header files]
|-- lib
|--- architecture (x86_64, arm64, ...)
|---- mode (shared or static)
|----- config (debug or release)
|------ [your libraries here in release mode]
|-- remaken-PackageName.pc
|-- packagedependencies.txt (if your Package requires third-parties) or packagedependencies-static.txt (if your package is in static build mode)
|-- PackageName-PackageVersion_remakeninfo.txt
|-- .pkginfo
|--- [.headers and/or .lib, and/or .bin folders]
|-- [... specific dependendy files] (wizards, csharp, xml for instance)
Remaken uses a packagedependencies.txt file defining all dependencies of your project. This file is interpreted by the QMake build script tools and is based on a simple dependency file (.pc).
Thus, by maintaining a simple dependency file and by using the remaken package tree describing the third parties used by your project, you will be able to download the dependencies, link and build your project, and install and deploy your solution with all its dependencies in a very simple way.
Let’s describe now how to package the third parties required by your project in order to be compliant with the dependency manager and the build and install the project.
To create your package, you have to create the the package tree structure described above.
This file is used by pkg-config to provide the necessary details for compilng and linking a program to a library.
If this file is not already provided with your third party, you will have to create it:
libname=PackageName
prefix=/usr/local
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/interfaces
Name: PackageName
Description: The PackageName library
Version: PackageVersion
Requires:
Libs: -L${libdir} -l${libname}
Libs.private: ${libdir}/${pfx}${libname}.${lext}
Cflags: -I${includedir}
For more information concerning the syntax of this pkg-config file, you can take a look to the pkg-config guide.
Remaken and builddefs-qmake support the recursive dependency download, link, and installation. But in order to do so, your package must precise its own dependencies in the packagedependencies.txt.
This file is specific to remaken and https://github.com/b-com-software-basis/builddefs-qmake, and define the platform, the C++ version and the runtime version used to build your package. This file is similar to the conaninfo.txt used by conan.
platform=win-cl-14.1
cppstd=17
runtime=dynamicCRT
When the previous files, your interfaces, your libraries and/or your binaries are ready and well-placed in the folder structure detailed above, you can package the dependency by using the ArtifactPackager scripts.
In scripts folder :
- win_artiPackager.bat
- unixes_artiPackager.sh
- mac_artiPackager.sh
Just run the ArtifactPackager script for current Os at the root folder of your package, and it will automatically create folder(s) for each version :
`architecture_mode_config` // for instance : `x86_64_shared_release`
Each folder contains a zip file with name format :
`PackageName_PackageVersion_architecture_mode_config.zip` // for instance : `xpcf_2.5.0_x86_64_shared_release.zip`
This package name format must be respected and will be used by remaken to find the dependency required for a requested development environment.
In order for Remaken to find the package corresponding to the requested dependency, you have to respect the following formatting rules concerning the URL to download your package:
URL_root/PackageName/PackageVersion/platform/PackageName_PackageVersion_architecture_mode_config.zip
Where:
- URL_root is the URL where you host your packages (github / artifactory ...)
- platform can be win, linux, mac, android, etc.
- Install pkg-config, cmake, conan
- Install msvc2019 community for build native desktop (Windows only)
- install a msvc2019 Qt kit for use qmake/jom (Windows only) or a gcc Qt kit (unixes)
from the scripts/unixes folder, run ./build_remaken.sh
from the scripts/win folder, run ./build_remaken.bat