Document editable packages

developing_packages.rst

@@ -9,5 +9,6 @@ This section describes how to work on packages which source code is being modifi
    :maxdepth: 2
 
    developing_packages/package_dev_flow
+   developing_packages/editable_packages
    developing_packages/workspaces
 

developing_packages/editable_packages.rst

@@ -0,0 +1,245 @@
+.. _editable_packages:
+
+Packages in editable mode
+=========================
+
+.. warning::
+
+    This is an **experimental** feature subject to breaking changes in future releases.
+
+When working in big projects with several functionalities interconnected it is recomended to avoid
+the one-and-only huge project approach in favor of several libraries each one of them specialized
+in a set of common tasks, even maintained by dedicated teams. This approach helps to isolate
+and reuse code, helps with compiling times and reduces the likelihood of including files that
+not correspond to the API of the required library.
+
+Nevertheless, in some case, it is useful to work in several libraries at the same time and see how
+the changes in one of them are propagated to the others. Following the
+:ref:`local workflow<package_dev_flow>` an user can execute the commands :command:`conan source`,
+:command:`conan install`, :command:`conan build` and :command:`conan package`, but in order to
+get the changes ready for a consumer library, it is needed the :command:`conan create` that will
+actually trigger a build to generate the binaries in the cache or to run :command:`conan export-pkg`
+to copy locally built artifacts into the conan cache and make them available to consumers.
+
+What about if you just can tell Conan where to find the headers and the artifacts ready for
+consumption in your local working directory? No need to package, just tell Conan to use those
+artifacts you have just generated with your IDE, sounds good? This is what the feature
+*editable packages* will do for you.
+
+Let's see this feature over an example where a developer is creating a ``CoolApp`` but at the same
+time they wants to work on ``cool/version@user/dev`` library which is tightly coupled to the app.
+
+The package ``cool/version@user/dev`` is already working, the developer has the sources in a
+local folder, they is using whatever method to build and develop locally and can perform
+a :command:`conan create . cool/version@user/dev` to create the package.
+
+Also, there is a *conanfile.txt* (or a more complex recipe) for the application ``CoolApp`` that
+has ``cool/version@user/dev`` among its requirements. When building this application, the
+resources of ``cool`` are used from the Conan local cache.
+
+Put a package in editable mode
+------------------------------
+
+To avoid creating the package ``cool/version@user/dev`` in the cache for every change, we are going
+to create **a link from the reference in the cache to the local working directory**:
+
+.. code-block:: bash
+
+    $ conan link <path/to/local/dev/libcool> cool/version@user/dev
+    # you could do cd <path/to/local/dev/libcool> && conan link . cool/version@user/dev
+
+
+That is it. Now, every usage of ``cool/version@user/dev``, by any other Conan package, or project
+will be redirected to the ``<path/to/local/dev/libcool>`` user folder, instead of using the package
+from the conan cache.
+
+Conan package recipes define a package "layout" in their ``package_info()`` methods. The default one,
+if nothing is specified is equivalent to:
+
+.. code-block:: python
+
+    def package_info(self):
+        # default behavior, doesn't need to be explicitly defined in recipes
+        self.cpp_info.includedirs = ["include"]
+        self.cpp_info.libdirs = ["lib"]
+        self.cpp_info.bindirs = ["bin"]
+        self.cpp_info.resdirs = ["res"]
+
+That means that conan will use the path ``path/to/local/dev/libcool/include`` for locating the headers of
+the ``cool`` package, the ``path/to/local/dev/libcool/lib`` to locate the libraries of the package, and so on.
+
+That might not be very useful, as typically while editing the source code and doing incremental builds, the
+development layout is different from that final "package" layout. While it is possible to run a
+:command:`conan package` local command to execute the packaging in the user folder, and that will achieve that
+final layout, that is not very elegant. Conan provides several ways to customize the layout for editable packages.
+
+Editable packages layouts
+-------------------------
+
+The custom layout of a package while it is in editable mode can be defined in different ways:
+
+Recipe defined layout
+++++++++++++++++++++++
+
+A recipe can defined a custom layout when it is not living in the local cache, in its ``package_info()`` method,
+something like:
+
+.. code-block:: python
+
+    from conans import ConanFile
+
+    class Pkg(ConanFile):
+        settings = "build_type"
+        def package_info(self):
+            if not self.in_local_cache:
+                d = "include_%s" % self.settings.build_type
+                self.cpp_info.includedirs = [d.lower()]
+
+That will map the include directories to ``path/to/local/dev/libcool/include_debug`` when working with ``build_type=Debug``
+conan setting, and to ``path/to/local/dev/libcool/include_release`` if ``build_type=Release``. In the same way, other 
+directories (libdirs, bindirs, etc) can be customized, with any logic, different for different OS, build systems, etc.
+
+.. code-block:: python
+
+    from conans import ConanFile
+
+    class Pkg(ConanFile):
+        settings = "os", "compiler", "arch", "build_type"
+        def package_info(self):
+            if not self.in_local_cache:
+                if self.settings.compiler == "Visual Studio":
+                    # NOTE: Use the real layout used in your VS projects, this is just an example
+                    self.cpp_info.libdirs = ["%s_%s" % (self.settings.build_type, self.settings.arch)]
+
+That will define the libraries directories to ``path/to/local/dev/libcool/Release_x86_64``, for example.
+That is only an example, the real layout used by VS would be different.
+
+Layout files
++++++++++++++
+
+Layout files have the following syntax:
+
+   .. code-block:: ini
+
+       [includedirs]
+       src/core/include
+       src/cmp_a/include
+
+       [libdirs]
+       build/{settings.build_type}/{settings.arch}
+
+       [bindirs]
+       build/{settings.build_type}/{settings.arch}
+
+As you can see, you can use some **placeholders** inside these files that will be substituted with
+the values of the ``settings`` and the ``options`` of the package.
+
+This file can use the package reference to customize logic for a specific package:
+
+   .. code-block:: ini
+
+       [includedirs]
+       src/include
+
+       [cool/version@user/dev:includedirs]
+       src/core/include
+
+This layout will define the ``src/core/include`` include directory for the ``cool`` package, and
+``src/include`` for other packages in editable mode.
+
+In every case the directories that will be affected by the editable mode will be ``includedirs``,
+``libdirs``, ``bindirs``, ``resdirs``, ``srcdirs`` and ``builddirs``, all of them declared in the
+:ref:`cpp_info_attributes_reference` dictionary; the rest of values in that dictionary won't
+be modified. So ``cflags``, ``defines``, library names in ``libs`` defined in ``package_info()``
+will still be used.
+
+By default all folders paths are relative to the directory where the *conanfile.py*
+of the editable package is (which is the path used to create the link), though they also allow absolute
+paths.
+
+Specifying layout files
++++++++++++++++++++++++
+
+Layout files are specified in the :command:`conan link` command, as an extra argument:
+
+.. code-block:: bash
+
+    $ conan link . cool/version@user/dev --layout=win_layout
+
+That ``win_layout`` file will be first looked for relative to the current directory (the
+path can be absolute too). If it is found, that will be used. It is possible to add those
+layouts in the source repositories, so they are always easy to find after a clone.
+
+If the specified layout is not found relative to the current directory, it will be looked
+for in the conan cache, in the ``.conan/layouts`` folder. This is very convenient to have
+a single definition of layouts that can be shared with the team and installed with
+``conan config install``.
+
+If no argument is specified, the :command:`conan link` command will try to use a `.conan/layouts/default`
+layout from the local cache.
+
+You can switch layout files by passing a different argument to new calls to :command:`conan link`.
+
+Evaluation order and priority
++++++++++++++++++++++++++++++
+
+It is important to understand the evaluation order and priorities regarding the definitions of layouts:
+
+- The first thing that will always execute is the recipe ``package_info()``. That will define
+  the flags, definitions, as well as some values for the layout folders: ``includedirs``, ``libdirs``, etc.
+- If a layout file is defined, either explicitly or using the implicit ``.conan/layouts/default``,
+  conan will look for matches, based on its package reference.
+- If a match is found, either because of global definitions like ``[includedirs]``
+  or because a match like ``[pkg/version@user/channel:includedirs]``, then the layout folders
+  (includedirs, libdirs, resdirs, builddirs, bindirs), will be invalidated and replaced by the ones
+  defined in the file.
+- If a specific match like ``[pkg/version@user/channel:includedirs]`` is found, it is expected to
+  have defined also its specific ``[pkg/version@user/channel:libdirs]``, etc. The global layout
+  folders specified without package reference won't be applied once a match is found.
+- It no match is found, the original values for the layout folders defined in ``package_info()`` will
+  be respected.
+- The layout file to be used is defined at ``conan link`` time. If a ``.conan/layouts/default`` file
+  is added after the ``conan link``, it will not be used at all.
+
+
+Using a package in editable mode
+--------------------------------
+
+Once a reference is in editable mode it is used **system wide** (for every set of ``settings`` and
+``options``) by Conan (by every Conan client that uses the same cache), no changes are
+required in the consumers. Every :command:`conan install` command that requires our editable
+``cool/version@user/dev`` package will use the paths to the local directory and the changes
+made to this project will be taken into account by the packages using its headers or linking
+against it. 
+
+To summarize, consumption of packages in editable mode is transparent to their consumers.
+To try that it is working, the following flow should work:
+
+- get sources of ``cool/version@user/dev``: :command:`git/svn clone... && cd folder`
+- Put package in editable mode: :command:`conan link . cool/version@user/dev --layout=mylayout`
+- Build it using the local flow: :command:`conan install` + build
+- Go to the consumer project: ``CoolApp``
+- Build it using the local flow: :command:`conan install` + build
+- Go back to ``cool/version@user/dev`` source folder, do some changes, and just build. No conan commands necessary
+- Go to the consumer project: ``CoolApp`` and rebuild. It should get the changes from the ``cool`` library.
+
+In that way, it is possible to be developing both the ``cool`` library and the ``CoolApp`` application, at the same
+time, without any Conan command.
+
+Revert the editable mode
+------------------------
+
+In order to revert the editable mode just remove the link using:
+
+.. code-block:: bash
+
+    $ conan link --remove cool/version@user/dev
+
+It will remove the link (the local directory won't be affected) and all the packages consuming this
+requirement will get it from the cache again.
+
+.. warning::
+
+   Packages that are built consuming an editable package in its graph upstreams can generate binaries
+   and packages incompatible with the released version of the editable package. Avoid uploading
+   these packages without re-creating them with the in-cache version of all the libraries.

reference/config_files.rst

@@ -15,4 +15,5 @@ These are the most important configuration files, used to customize conan.
    config_files/registry.txt
    config_files/client_certificates
    config_files/artifacts.properties
+   config_files/editable_layout
 

reference/config_files/editable_layout.rst

@@ -0,0 +1,41 @@
+.. _editable_layout:
+
+Editable layout files
+=====================
+
+This file contain information consumed by :ref:`editable packages <editable_packages>`. It is
+an *.ini* file listing the directories that Conan should use for the packages that are opened
+in editable mode:
+
+.. code-block:: ini
+
+    # Affects to all packages but cool/version@user/dev
+    [includedirs]
+    src/include
+
+    # using placeholders from conan settings and options
+    [libdirs]
+    build/{settings.build_type}/{settings.arch}
+
+    [bindirs]
+    build/{settings.build_type}/{settings.arch}
+
+    # Affects only to cool/version@user/dev
+    [cool/version@user/dev:includedirs]
+    src/core/include
+    src/cmp_a/include
+
+
+The specific sections using a package reference will have higher priority than the general ones.
+
+
+This file can live in the conan cache, in the ``.conan/layouts`` folder, or in a user folder, like
+inside the source repo.
+
+If there exists a ``.conan/layouts/default`` layout file in the cache and no layout file is specified
+in the ``conan link <path> <reference>`` command, that file will be used.
+
+
+.. seealso::
+
+    Check the section :ref:`editable_packages` to read more about this file.