Add documentation about workspaces, including overriding packages (#88)

* Rough progress

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Small rst fixups

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Stubs for other known issues

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* MOre text for ABI/API incompatibility

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Add table of contents

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Fix ToC

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Fix list indentation

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Document static linking issue

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Shallower TOC

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Start with tips on avoiding overriding and fix headings

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Result of more refactoring

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* More doc updates

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* more fleshing out documentation

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Alter example to make transitive issue clear

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* RST syntax

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* more updates

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* More updates

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Simplify

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Misc changes from proofreading

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Rename file

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* ROS binary packages are a merged workspace

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* compaired -> compared

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Jacob Perron <jacob@openrobotics.org>

* Shorten sentence

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* Shorten sentence

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* entrypoints -> entry points

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* overridding -> overriding

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* etc -> etc.

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* most -> common

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* most -> common

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* most -> common

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* it's -> it is

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* add to

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* Fix singular/plural usage

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* clearer description of workspaces in example

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>

* WIP explain what a workspace is in a tutorial

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Formatting fixes

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Simpler test

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* More progress

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* first draft of what is a workspace done

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* node -> note

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* First draft of isolated-vs-merged workspaces

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Add documentation on using multiple workspaces

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Link to isolated-vs-merged workspaces

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Move isolated workspace doc link

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Reword install headers to unique directory example

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* node -> note

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Add workspaces to index

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* setup.cfg in src folder

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Describe stdout_stderr.log

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Describe N workspace soruing terms after two workspace terms

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* misc, including folder->directory

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Clearer workspace root sentence

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* List consisten with sections

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* shorten sentence

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* directorys -> directories

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Clearer sentences about software packages

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* typo

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Remove confusing comment

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Full test -> The test's

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Misc grammar

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Combine top two paragraphs

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Fix nested list rendering

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Promote heading

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Capitalization

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Conclusion isn't really necessary

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* Conclusion isn't really necessary

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

* point -> points

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Chris Lalancette <clalancette@gmail.com>

* No period in bullet list

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Chris Lalancette <clalancette@gmail.com>

* No period in bullet list

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Chris Lalancette <clalancette@gmail.com>

* space in rst link

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Chris Lalancette <clalancette@gmail.com>

* such as Windows 10

Signed-off-by: Shane Loretz <sloretz@osrfoundation.org>

Co-authored-by: Chris Lalancette <clalancette@gmail.com>

* Touch setup.cfg

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* Remove duplicate note about sourcing in different terminal

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* link to isolated vs merged workspaces page

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* clean install wording less ambiguous

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* Describe sorting headers more generally

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* Clearer sentences about recompiling

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* A little more context on ROS 1 feature

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* mentioned -> aforementioned

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Scott K Logan <logans@cottsay.net>

* clarify another package finds the headers

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Scott K Logan <logans@cottsay.net>

* overriding -> override

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Scott K Logan <logans@cottsay.net>

* insert 'to'

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Scott K Logan <logans@cottsay.net>

* numger -> number

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Scott K Logan <logans@cottsay.net>

* describe package as having been built rather than existing

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Scott K Logan <logans@cottsay.net>

* relax environemtn variable length description

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Scott K Logan <logans@cottsay.net>

* Assume colcon-common-extensions

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* simplify sentence

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* Add page describing log files in more detail

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* Add missing comma

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Scott K Logan <logans@cottsay.net>

* Sourcing a workspace allows building against it

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* Remove inconsistent platform library extensions

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* Undefined -> unpredictable

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* Describe local_setup.*

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

* depdencencies -> dependencies

Signed-off-by: Shane Loretz <sloretz@openrobotics.org>

Co-authored-by: Chris Lalancette <clalancette@gmail.com>

Co-authored-by: Geoffrey Biggs <gbiggs@killbots.net>
Co-authored-by: Jacob Perron <jacob@openrobotics.org>
Co-authored-by: Chris Lalancette <clalancette@gmail.com>
Co-authored-by: Scott K Logan <logans@cottsay.net>

index.rst

@@ -34,6 +34,11 @@ Information about development is also available:
    user/quick-start
    user/configuration
    user/how-to
+   user/what-is-a-workspace
+   user/log-files
+   user/isolated-vs-merged-workspaces
+   user/using-multiple-workspaces
+   user/overriding-packages
 
 .. _reference-docs:
 

user/how-to.rst

@@ -173,31 +173,3 @@ For debugging purposes you can enable logging messages with other levels (e.g. `
 .. code-block:: bash
 
     $ colcon --log-level info <verb> ...
-
-Log files of past invocations
------------------------------
-
-By default the ``log`` directory is created as a sibling to the ``src`` directory.
-Some verbs (e.g. ``build``, ``test``, ``test-result``) generate log files in a subdirectory which is named following the pattern ``<verb>_<timestamp>``.
-For the latest invocation of a specific verb there is a symlink named ``latest_<verb>`` (on platforms which support symbolic links).
-For the latest invocation there is another symlink just named ``latest`` (on platforms which support symbolic links).
-
-Each log directory contains a couple of files in the root:
-
-* ``events.log`` contains all internal events dispatched.
-  This file is mostly for debugging purposes.
-* ``logger_all.log`` contains all logging messages even though the invocation didn't show them on the console.
-  This is helpful to see log message with a different level after a command was run.
-  The first line of this file contains the exact command line invocation including all the arguments passed.
-
-For each package additional files are being created in a subdirectory named after the package:
-
-* ``command.log`` contains the commands which have been invoked for the package, e.g. calls to ``python setup.py``.
-* ``stdout.log`` contains all the output the invoked commands printed to ``stdout``.
-* ``stderr.log`` contains all the output the invoked commands printed to ``stderr``.
-* ``stdout_stderr.log`` contains all the output the invoked commands printed to either of the two pipes in the order they appeared.
-* ``streams.log`` combines the output of all the other log files in the order they appeared.
-
-.. note::
-
-    While ``colcon`` is doing its best to read concurrently from the ``stdout`` and ``stderr`` pipes to preserve the order of output it can't guarantee the correctness of the order in all cases.

user/isolated-vs-merged-workspaces.rst

@@ -0,0 +1,73 @@
+Isolated vs Merged Workspaces
+=============================
+
+Assuming you know :doc:`what a workspace is <what-is-a-workspace>`, you know that by default colcon builds an **isolated workspace**.
+It can also build a **merged workspace**.
+These terms describe the layout of the **install space**, which is the directory software gets installed into.
+
+This page describes the differences between the two.
+
+
+Isolated Workspace
+------------------
+
+Colcon by default builds an isolated workspace.
+
+Build the workspace from :doc:`what-is-a-workspace` without any extra arguments.
+
+.. code-block:: bash
+
+	colcon build
+
+Now let's look in the ``install`` folder.
+
+::
+
+	install
+	├── COLCON_IGNORE
+	├── foo
+	│	├── lib/...
+	│	└── share/...
+	├── local_setup.[bash|bat|ps1|sh|zsh|...]
+	├── _local_setup_util_[sh|ps1|...].py
+	└── setup.[bash|bat|ps1|sh|zsh|...]
+
+Colcon created a directory ``install/foo`` and installed the package ``foo`` inside of it.
+Building an isolated workspace just means every software package is installed into its own directory.
+
+An isolated workspace has some advantages.
+When colcon tests a package in an isolated workspace it will only give the tests access to the install artifacts of the dependencies it declares.
+This allows users to catch undeclared dependencies.
+An isolated workspace also allows removing a single package's install artifacts by deleting its directory.
+
+
+Merged Workspace
+----------------
+
+Delete the ``install`` directory and build the workspace again with the ``--merge-install`` option.
+
+.. code-block:: bash
+
+	colcon build --merge-install
+
+
+Let's look in the ``install`` folder again.
+
+::
+
+	install
+		├── COLCON_IGNORE
+		├── lib/..
+		├── share/..
+		├── local_setup.[bash|bat|ps1|sh|zsh|...]
+		├── _local_setup_util_[sh|ps1|...].py
+		└── setup.[bash|bat|ps1|sh|zsh|...]
+
+
+Notice how there's no longer a ``foo`` folder. 
+The ``lib`` and ``share`` directories are now directly in ``install``.
+
+Building a merged workspace just means every software package is installed into the same directory.
+
+A merged workspace is advantageous on platforms where environment variables length is more tightly constrained, such as Windows 10.
+Environment variables set by the shell scripts, such as ``PYTHONPATH``, will be shorter because there only needs to be one entry for all of the installed packages.

user/log-files.rst

@@ -0,0 +1,68 @@
+Log Files
+=========
+
+You may notice with ``colcon-common-extensions`` installed there's not much output to the console when you run ``colcon build`` or ``colcon test``.
+The log output isn't lost; it was written to the ``log`` directory.
+
+.. note::
+
+    The ``--event-handlers`` argument can be used to output build logs to the console.
+    For example, ``colcon build --event-handlers console_direct+`` or ``colcon test --event-handlers console_direct+`` will output everything in real time.
+
+Let's look at the ``log`` directory from :doc:`what-is-a-workspace`.
+
+::
+
+  log
+  ├── build_2022-05-20_11-50-03
+  │ ├── events.log
+  │ ├── foo
+  │ │ ├── command.log
+  │ │ ├── stderr.log
+  │ │ ├── stdout.log
+  │ │ ├── stdout_stderr.log
+  │ │ └── streams.log
+  │ └── logger_all.log
+  ├── COLCON_IGNORE
+  ├── latest -> latest_build
+  ├── latest_build -> build_2022-05-20_11-50-03
+  ├── latest_test -> test_2022-05-20_11-50-05
+  └── test_2022-05-20_11-50-05
+      ├── events.log
+      ├── foo
+      │ ├── command.log
+      │ ├── stderr.log
+      │ ├── stdout.log
+      │ ├── stdout_stderr.log
+      │ └── streams.log
+      └── logger_all.log
+
+
+The directory ``build_<date and time>`` contains all logs from the invocation of ``colcon build``.
+``test_<date and time>`` contains all the logs from the invocation of ``colcon test``.
+On platforms that support symbolic links, ``latest_build`` points to the most recent build, and ``latest_test`` does the same for tests.
+
+More generally, any verb that generates logs (e.g. ``build``, ``test``, ``test-result``) will put them into a subdirectory named ``<verb>_<date and time>``.
+``latest_<verb>`` will point to the logs of the most recent invocation of that verb.
+The symbolic link ``latest`` will point to the most recent logs for any colcon verb.
+
+Each ``<verb>_<date and time>`` directory has these files:
+
+* ``events.log`` contains all internal events dispatched.
+  This file is mostly for debugging purposes.
+* ``logger_all.log`` contains all logging messages regardless of the log level.
+  The first line of this file contains the exact command line invocation including all the arguments passed.
+
+Notice each ``<verb>_<date and time>`` directory has a ``foo`` directory.
+More generally, a directory for every package is created.
+Package log directories have these files:
+
+* ``command.log`` contains the commands invoked for the package, e.g. calls to ``python setup.py``.
+* ``stdout.log`` contains all the output printed to ``stdout``.
+* ``stderr.log`` contains all the output printed to ``stderr``.
+* ``stdout_stderr.log`` contains all the output printed to either ``stdout`` or ``stderr`` in the order they appeared.
+* ``streams.log`` combines the output of all log files in the order they appeared.
+
+.. note::
+
+    ``colcon`` does its best to read concurrently from ``stdout`` and ``stderr`` to preserve the order, but it can't guarantee correct order of log messages in all cases.