From 4d8b82e70d68094423046c53239c551d5d344f85 Mon Sep 17 00:00:00 2001 From: Carlos Zoido Date: Thu, 16 Feb 2023 08:44:46 +0100 Subject: [PATCH] Tutorial for local flow (#2968) * wip * wip * wip * wip * wip * fix spelling * Update tutorial/developing_packages/local_package_development_flow.rst Co-authored-by: James * fix format * remove test * add note --------- Co-authored-by: James --- .../layout/third_party_libraries.rst | 2 +- reference/commands.rst | 42 ++- reference/commands/build.rst | 2 + reference/commands/download.rst | 31 ++ reference/commands/editable.rst | 24 ++ reference/commands/export-pkg.rst | 67 ++++ reference/commands/source.rst | 25 ++ reference/commands/test.rst | 2 + .../add_dependencies_to_packages.rst | 2 +- tutorial/creating_packages/build_packages.rst | 2 +- .../define_package_information.rst | 2 +- .../package_prebuilt_binaries.rst | 2 + .../creating_packages/preparing_the_build.rst | 2 +- .../local_package_development_flow.rst | 296 +++++++++++++++++- 14 files changed, 472 insertions(+), 29 deletions(-) create mode 100644 reference/commands/download.rst create mode 100644 reference/commands/editable.rst create mode 100644 reference/commands/export-pkg.rst create mode 100644 reference/commands/source.rst diff --git a/examples/conanfile/layout/third_party_libraries.rst b/examples/conanfile/layout/third_party_libraries.rst index 5c0c5779551..0f881b99c64 100644 --- a/examples/conanfile/layout/third_party_libraries.rst +++ b/examples/conanfile/layout/third_party_libraries.rst @@ -45,7 +45,7 @@ The ``conanfile.py`` would look like this: # the downloaded soures will be inside the "src" subfolder get(self, "https://github.com/conan-io/libhello/archive/refs/heads/main.zip", strip_root=True) - # Please, be aware that using the head of the branch instead of an inmutable tag + # Please, be aware that using the head of the branch instead of an immutable tag # or commit is not a good practice in general as the branch may change the contents # patching, replacing, happens here diff --git a/reference/commands.rst b/reference/commands.rst index a8280c7c6f3..430800bdaf6 100644 --- a/reference/commands.rst +++ b/reference/commands.rst @@ -43,31 +43,29 @@ and these :ref:`custom command examples ` **Creator commands:** -.. toctree:: - :caption: Creator commands - :maxdepth: 1 - :hidden: - - commands/create - commands/export - commands/new - commands/test - commands/upload +.. toctree:: + :caption: Creator commands + :maxdepth: 1 + :hidden: + + commands/build + commands/create + commands/download + commands/editable + commands/export + commands/export-pkg + commands/new + commands/source + commands/test + commands/upload +- :doc:`conan build `: Install package and call its build method - :doc:`conan create `: Create a package from a recipe +- :doc:`conan download `: Download (without install) a single conan package from a remote server. +- :doc:`conan editable `: Allows working with a package in user folder - :doc:`conan export `: Export a recipe to the Conan package cache +- :doc:`conan export-pkg `: Create a package directly from pre-compiled binaries - :doc:`conan new `: Create a new recipe from a predefined template +- :doc:`conan source `: Calls the source() method - :doc:`conan test `: Test a package - :doc:`conan upload `: Upload packages from the local cache to a specified remote - - -**Development commands** - -.. toctree:: - :caption: Development commands - :maxdepth: 1 - :hidden: - - commands/build - -- :doc:`conan build `: Install package and call its build method diff --git a/reference/commands/build.rst b/reference/commands/build.rst index 437129d6b54..028dc1c8e95 100644 --- a/reference/commands/build.rst +++ b/reference/commands/build.rst @@ -1,3 +1,5 @@ +.. _reference_commands_build: + conan build =========== diff --git a/reference/commands/download.rst b/reference/commands/download.rst new file mode 100644 index 00000000000..10b18774719 --- /dev/null +++ b/reference/commands/download.rst @@ -0,0 +1,31 @@ +.. _reference_commands_download: + +conan download +============== + +.. code-block:: bash + + $ conan download --help + usage: conan download [-h] [-v [V]] [--logger] [--only-recipe] [-p PACKAGE_QUERY] -r REMOTE reference + + Download (without install) a single conan package from a remote server. + + It downloads just the package, but not its transitive dependencies, and it will not call + any generate, generators or deployers. + It can download multiple packages if patterns are used, and also queries over the package + binaries can be provided. + + positional arguments: + reference Recipe reference or package reference, can contain * as wildcard at any reference field. If + revision is not specified, it is assumed latest one. + + optional arguments: + -h, --help show this help message and exit + -v [V] Level of detail of the output. Valid options from less verbose to more verbose: -vquiet, -verror, + -vwarning, -vnotice, -vstatus, -v or -vverbose, -vv or -vdebug, -vvv or -vtrace + --logger Show the output with log format, with time, type and message. + --only-recipe Download only the recipe/s, not the binary packages. + -p PACKAGE_QUERY, --package-query PACKAGE_QUERY + Only upload packages matching a specific query. e.g: os=Windows AND (arch=x86 OR compiler=gcc) + -r REMOTE, --remote REMOTE + Download from this specific remote diff --git a/reference/commands/editable.rst b/reference/commands/editable.rst new file mode 100644 index 00000000000..8a3cdfe7724 --- /dev/null +++ b/reference/commands/editable.rst @@ -0,0 +1,24 @@ +.. _reference_commands_editable: + +conan editable +============== + +.. code-block:: bash + + $ conan editable --help + usage: conan editable [-h] [-v [V]] [--logger] {add,list,remove} ... + + Allows working with a package in user folder + + positional arguments: + {add,list,remove} sub-command help + add Define the given location as the package , so when this package is required, it is + used from this location instead of from the cache + list List packages in editable mode + remove Remove the "editable" mode for this reference. + + optional arguments: + -h, --help show this help message and exit + -v [V] Level of detail of the output. Valid options from less verbose to more verbose: -vquiet, -verror, + -vwarning, -vnotice, -vstatus, -v or -vverbose, -vv or -vdebug, -vvv or -vtrace + --logger Show the output with log format, with time, type and message. diff --git a/reference/commands/export-pkg.rst b/reference/commands/export-pkg.rst new file mode 100644 index 00000000000..97a083414c8 --- /dev/null +++ b/reference/commands/export-pkg.rst @@ -0,0 +1,67 @@ +.. _reference_commands_export-pkg: + +conan export-pkg +================ + +.. code-block:: bash + + $ conan export-pkg --help + usage: conan export-pkg [-h] [-f FORMAT] [-v [V]] [--logger] [-of OUTPUT_FOLDER] [--name NAME] [--version VERSION] + [--user USER] [--channel CHANNEL] [-l LOCKFILE] [--lockfile-partial] [--lockfile-out LOCKFILE_OUT] + [--lockfile-packages] [--lockfile-clean] [-o OPTIONS_HOST] [-o:b OPTIONS_BUILD] [-o:h OPTIONS_HOST] + [-pr PROFILE_HOST] [-pr:b PROFILE_BUILD] [-pr:h PROFILE_HOST] [-s SETTINGS_HOST] + [-s:b SETTINGS_BUILD] [-s:h SETTINGS_HOST] [-c CONF_HOST] [-c:b CONF_BUILD] [-c:h CONF_HOST] + path + + Create a package directly from pre-compiled binaries + + positional arguments: + path Path to a folder containing a recipe (conanfile.py) + + optional arguments: + -h, --help show this help message and exit + -f FORMAT, --format FORMAT + Select the output format: json + -v [V] Level of detail of the output. Valid options from less verbose to more verbose: -vquiet, -verror, + -vwarning, -vnotice, -vstatus, -v or -vverbose, -vv or -vdebug, -vvv or -vtrace + --logger Show the output with log format, with time, type and message. + -of OUTPUT_FOLDER, --output-folder OUTPUT_FOLDER + The root output folder for generated and build files + --name NAME Provide a package name if not specified in conanfile + --version VERSION Provide a package version if not specified in conanfile + --user USER Provide a user if not specified in conanfile + --channel CHANNEL Provide a channel if not specified in conanfil + -l LOCKFILE, --lockfile LOCKFILE + Path to a lockfile. Use --lockfile="" to avoid automatic use of existing 'conan.lock' file + --lockfile-partial Do not raise an error if some dependency is not found in lockfile + --lockfile-out LOCKFILE_OUT + Filename of the updated lockfile + --lockfile-packages Lock package-id and package-revision information + --lockfile-clean remove unused + -o OPTIONS_HOST, --options OPTIONS_HOST + Define options values (host machine), e.g.: -o Pkg:with_qt=true + -o:b OPTIONS_BUILD, --options:build OPTIONS_BUILD + Define options values (build machine), e.g.: -o:b Pkg:with_qt=true + -o:h OPTIONS_HOST, --options:host OPTIONS_HOST + Define options values (host machine), e.g.: -o:h Pkg:with_qt=true + -pr PROFILE_HOST, --profile PROFILE_HOST + Apply the specified profile to the host machine + -pr:b PROFILE_BUILD, --profile:build PROFILE_BUILD + Apply the specified profile to the build machine + -pr:h PROFILE_HOST, --profile:host PROFILE_HOST + Apply the specified profile to the host machine + -s SETTINGS_HOST, --settings SETTINGS_HOST + Settings to build the package, overwriting the defaults (host machine). e.g.: -s compiler=gcc + -s:b SETTINGS_BUILD, --settings:build SETTINGS_BUILD + Settings to build the package, overwriting the defaults (build machine). e.g.: -s:b compiler=gcc + -s:h SETTINGS_HOST, --settings:host SETTINGS_HOST + Settings to build the package, overwriting the defaults (host machine). e.g.: -s:h compiler=gcc + -c CONF_HOST, --conf CONF_HOST + Configuration to build the package, overwriting the defaults (host machine). e.g.: -c + tools.cmake.cmaketoolchain:generator=Xcode + -c:b CONF_BUILD, --conf:build CONF_BUILD + Configuration to build the package, overwriting the defaults (build machine). e.g.: -c:b + tools.cmake.cmaketoolchain:generator=Xcode + -c:h CONF_HOST, --conf:host CONF_HOST + Configuration to build the package, overwriting the defaults (host machine). e.g.: -c:h + tools.cmake.cmaketoolchain:generator=Xcode diff --git a/reference/commands/source.rst b/reference/commands/source.rst new file mode 100644 index 00000000000..7cfad5da404 --- /dev/null +++ b/reference/commands/source.rst @@ -0,0 +1,25 @@ +.. _reference_commands_source: + +conan source +============ + +.. code-block:: bash + + $ conan source --help + usage: conan source [-h] [-v [V]] [--logger] [--name NAME] [--version VERSION] [--user USER] [--channel CHANNEL] [path] + + Calls the source() method + + positional arguments: + path Path to a folder containing a recipe (conanfile.py or conanfile.txt) or to a recipe file. e.g., + ./my_project/conanfile.txt. + + optional arguments: + -h, --help show this help message and exit + -v [V] Level of detail of the output. Valid options from less verbose to more verbose: -vquiet, -verror, + -vwarning, -vnotice, -vstatus, -v or -vverbose, -vv or -vdebug, -vvv or -vtrace + --logger Show the output with log format, with time, type and message. + --name NAME Provide a package name if not specified in conanfile + --version VERSION Provide a package version if not specified in conanfile + --user USER Provide a user if not specified in conanfile + --channel CHANNEL Provide a channel if not specified in conanfile diff --git a/reference/commands/test.rst b/reference/commands/test.rst index 30017280f5c..5512face51e 100644 --- a/reference/commands/test.rst +++ b/reference/commands/test.rst @@ -1,3 +1,5 @@ +.. _reference_commands_test: + conan test =========== diff --git a/tutorial/creating_packages/add_dependencies_to_packages.rst b/tutorial/creating_packages/add_dependencies_to_packages.rst index 4b897ad132d..a8597de1a60 100644 --- a/tutorial/creating_packages/add_dependencies_to_packages.rst +++ b/tutorial/creating_packages/add_dependencies_to_packages.rst @@ -47,7 +47,7 @@ Let's check the relevant parts: def source(self): git = Git(self) git.clone(url="https://github.com/conan-io/libhello.git", target=".") - # Please, be aware that using the head of the branch instead of an inmutable tag + # Please, be aware that using the head of the branch instead of an immutable tag # or commit is not a good practice in general git.checkout("require_fmt") diff --git a/tutorial/creating_packages/build_packages.rst b/tutorial/creating_packages/build_packages.rst index 5710b6345bd..623c129aa75 100644 --- a/tutorial/creating_packages/build_packages.rst +++ b/tutorial/creating_packages/build_packages.rst @@ -42,7 +42,7 @@ Changes introduced in the recipe def source(self): git = Git(self) git.clone(url="https://github.com/conan-io/libhello.git", target=".") - # Please, be aware that using the head of the branch instead of an inmutable tag + # Please, be aware that using the head of the branch instead of an immutable tag # or commit is not a good practice in general git.checkout("with_tests") diff --git a/tutorial/creating_packages/define_package_information.rst b/tutorial/creating_packages/define_package_information.rst index bbc06f10aa8..62e24966b29 100644 --- a/tutorial/creating_packages/define_package_information.rst +++ b/tutorial/creating_packages/define_package_information.rst @@ -159,7 +159,7 @@ conditionally set the library name depending on the ``self.options.shared`` opti def source(self): git = Git(self) git.clone(url="https://github.com/conan-io/libhello.git", target=".") - # Please, be aware that using the head of the branch instead of an inmutable tag + # Please, be aware that using the head of the branch instead of an immutable tag # or commit is not a good practice in general git.checkout("package_info") diff --git a/tutorial/creating_packages/other_types_of_packages/package_prebuilt_binaries.rst b/tutorial/creating_packages/other_types_of_packages/package_prebuilt_binaries.rst index ebf29fa9aec..08d030eab72 100644 --- a/tutorial/creating_packages/other_types_of_packages/package_prebuilt_binaries.rst +++ b/tutorial/creating_packages/other_types_of_packages/package_prebuilt_binaries.rst @@ -1,3 +1,5 @@ +.. _creating_packages_other_prebuilt: + Package prebuilt binaries ========================= diff --git a/tutorial/creating_packages/preparing_the_build.rst b/tutorial/creating_packages/preparing_the_build.rst index 6f3e5b52d1b..b9342b3c879 100644 --- a/tutorial/creating_packages/preparing_the_build.rst +++ b/tutorial/creating_packages/preparing_the_build.rst @@ -64,7 +64,7 @@ Let's check the relevant parts: def source(self): git = Git(self) git.clone(url="https://github.com/conan-io/libhello.git", target=".") - # Please, be aware that using the head of the branch instead of an inmutable tag + # Please, be aware that using the head of the branch instead of an immutable tag # or commit is not a good practice in general git.checkout("optional_fmt") diff --git a/tutorial/developing_packages/local_package_development_flow.rst b/tutorial/developing_packages/local_package_development_flow.rst index 1abb97ef899..e57ed0fe99f 100644 --- a/tutorial/developing_packages/local_package_development_flow.rst +++ b/tutorial/developing_packages/local_package_development_flow.rst @@ -1,4 +1,296 @@ .. _local_package_development_flow: -Package development flow -========================= +Package Development Flow +======================== + +This section introduces the **Conan local development flow**, which allows you to work on +packages in your local project directory without having to export the contents of the +package to the Conan cache first. + +This local workflow encourages users to perform trial-and-error in a local sub-directory +relative to their recipe, much like how developers typically test building their projects +with other build tools. The strategy is to test the `conanfile.py` methods individually +during this phase. + +Let's use this flow for the ``hello`` package we created in :ref:`the previous +section`. + +Please clone the sources to recreate this project. You can find them in the `examples2.0 +repository `_ on GitHub: + +.. code-block:: bash + + $ git clone https://github.com/conan-io/examples2.git + $ cd examples2/tutorial/developing_packages/local_package_development_flow + +You can check the contents of the folder: + +.. code-block:: text + + . + ├── conanfile.py + └── test_package + ├── CMakeLists.txt + ├── conanfile.py + └── src + └── example.cpp + +conan source +------------ + +You will generally want to start with the :command:`conan source` command. The strategy +here is that you’re testing your source method in isolation and downloading the files to a +temporary sub-folder relative to the `conanfile.py`. This relative folder is defined by +the `self.folders.source` property in the `layout()` method. In this case, as we are using +the pre-defined `cmake_layout` we set the value with the `src_folder` argument. + +.. note:: + + In this example we are packaging a third-party library from a remote repository. In the + case you have your sources beside your recipe in the same repository, running + :command:`conan source` will not be necessary for most of the cases. + +Let's have a look at the recipe's `source()` and `layout()` method: + +.. code-block:: python + + ... + + def source(self): + # Please be aware that using the head of the branch instead of an immutable tag + # or commit is not a good practice in general. + get(self, "https://github.com/conan-io/libhello/archive/refs/heads/main.zip", + strip_root=True) + + def layout(self): + cmake_layout(self, src_folder="src") + + ... + + +Now run the :command:`conan source` command and check the results: + +.. code-block:: bash + + $ conan source . + conanfile.py (hello/1.0): Calling source() in /Users/.../local_package_development_flow/src + Downloading main.zip + conanfile.py (hello/1.0): Unzipping 3.7KB + Unzipping 100% + +You can see that a new `src` folder has appeared containing all the `hello` library sources. + +.. code-block:: text + :emphasize-lines: 3-10 + + . + ├── conanfile.py + ├── src + │ ├── CMakeLists.txt + │ ├── LICENSE + │ ├── README.md + │ ├── include + │ │ └── hello.h + │ └── src + │ └── hello.cpp + └── test_package + ├── CMakeLists.txt + ├── conanfile.py + └── src + └── example.cpp + +Now it's easy to check the sources and validate them. Once you've got your source method +right and it contains the files you expect, you can move on to testing the various +attributes and methods related to downloading dependencies. + +conan install +------------- + +After running the :command:`conan source` command, you can run the :command:`conan +install` command. This command will install all the recipe requirements if needed and +prepare all the files necessary for building by running the ``generate()`` method. + +We can check all the parts from our recipe that are involved in this step: + +.. code-block:: python + + ... + + class helloRecipe(ConanFile): + + ... + + generators = "CMakeDeps" + + ... + + def layout(self): + cmake_layout(self, src_folder="src") + + def generate(self): + tc = CMakeToolchain(self) + tc.generate() + + ... + +Now run the :command:`conan install` command and check the results: + +.. code-block:: bash + + $ conan install . + ... + -------- Finalizing install (deploy, generators) -------- + conanfile.py (hello/1.0): Writing generators to ... + conanfile.py (hello/1.0): Generator 'CMakeDeps' calling 'generate()' + conanfile.py (hello/1.0): Calling generate() + ... + conanfile.py (hello/1.0): Aggregating env generators + +You can see that a new `build` folder appeared with all the files that Conan needs for +building the library like a toolchain for `CMake` and several environment configuration +files. + +.. code-block:: text + :emphasize-lines: 3-10 + + . + ├── build + │ └── Release + │ └── generators + │ ├── CMakePresets.json + │ ├── cmakedeps_macros.cmake + │ ├── conan_toolchain.cmake + │ ├── conanbuild.sh + │ ├── conanbuildenv-release-x86_64.sh + │ ├── conanrun.sh + │ ├── conanrunenv-release-x86_64.sh + │ ├── deactivate_conanbuild.sh + │ └── deactivate_conanrun.sh + ├── conanfile.py + ├── src + │ ├── CMakeLists.txt + │ ├── CMakeUserPresets.json + │ ├── LICENSE + │ ├── README.md + │ ├── include + │ │ └── hello.h + │ └── src + │ └── hello.cpp + └── test_package + ├── CMakeLists.txt + ├── conanfile.py + └── src + └── example.cpp + +Now that all the files necessary for building are generated, you can move on to testing +the `build()` method. + +conan build +----------- + +Running the After :command:`conan build` command will invoke the `build()` method: + +.. code-block:: python + + ... + + class helloRecipe(ConanFile): + + ... + + def build(self): + cmake = CMake(self) + cmake.configure() + cmake.build() + + ... + +Let's run :command:`conan build`: + +.. code-block:: bash + + $ conan build . + ... + -- Conan toolchain: C++ Standard 11 with extensions ON + -- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF + -- Configuring done + -- Generating done + -- Build files have been ... + conanfile.py (hello/1.0): CMake command: cmake --build ... + conanfile.py (hello/1.0): RUN: cmake --build ... + [100%] Built target hello + +For most of the recipes, the `build()` method should be very simple, and you can also +invoke the build system directly, without invoking Conan, as you have all the necessary +files available for building. If you check the contents of the `src` folder, you'll find a +`CMakeUserPresets.json` file that you can use to configure and build the `conan-release` +preset. Let's try it: + +.. code-block:: bash + + $ cd src + $ cmake --preset conan-release + ... + -- Configuring done + -- Generating done + + $ cmake --build --preset conan-release + ... + [100%] Built target hello + +You can check that the results of invoking CMake directly are equivalent to the ones we +got using the :command:`conan build` command. + +.. include:: ../cmake_presets_note.rst + +conan export-pkg +---------------- + +Now that we built the package binaries locally we can also package those artifacts in the +Conan local cache using the :command:`conan export-pkg` command. Please note that this +command will create the package in the Conan cache and test it running the `test_package` +after that. + +.. code-block:: bash + + $ conan export-pkg . + conanfile.py (hello/1.0) package(): Packaged 1 '.h' file: hello.h + conanfile.py (hello/1.0) package(): Packaged 1 '.a' file: libhello.a + conanfile.py (hello/1.0): Package 'b1d267f77ddd5d10d06d2ecf5a6bc433fbb7eeed' created + conanfile.py (hello/1.0): Created package revision f09ef573c22f3919ba26ee91ae444eaa + ... + conanfile.py (hello/1.0): Package folder /Users/... + conanfile.py (hello/1.0): Exported package binary + ... + [ 50%] Building CXX object CMakeFiles/example.dir/src/example.cpp.o + [100%] Linking CXX executable example + [100%] Built target example + + -------- Testing the package: Running test() -------- + hello/1.0 (test package): Running test() + hello/1.0 (test package): RUN: ./example + hello/1.0: Hello World Release! + hello/1.0: __x86_64__ defined + hello/1.0: __cplusplus201103 + hello/1.0: __GNUC__4 + hello/1.0: __GNUC_MINOR__2 + hello/1.0: __clang_major__14 + hello/1.0: __apple_build_version__14000029 + +Now you can list the packages in the local cache and check that the ``hello/1.0`` package +was created. + +.. code-block:: bash + + $ conan list hello/1.0 + Local Cache + hello + hello/1.0 + +.. seealso:: + + - Reference for conan :ref:`source`, + :ref:`install`, :ref:`build`, + :ref:`export-pkg` and + :ref:`test` commands. + - Packaging prebuilt binaries :ref:`example`