CI is stuck on virtualenv 16.7.9 until install-scripts option is honored.

[nuclearsandwich]

ROS 2 Python packages which provide executables intended to be run via ros2 run $PACKAGENAME $EXENAME use the [install] install-scripts option of setup.cfg to install binaries into $base/lib/$PACKAGENAME, e.g. $base/lib/demo_nodes_py for the demo_nodes_py package. This intended behavior is thwarted when using virtualenv>20 or the venv module to create the virtual environment.

setuptools (and distutils) have code in place to detect when they're operating in a virtualenv and ignore installation-related options from setup.cfg entirely 1. The options are ignored whenever sys.prefix != sys.base_prefix. When using virtualenv<20, sys.prefix and sys.base_prefix both point to the virtualenv prefix. Using the venv module or virtualenv>=20, sys.base_prefix is the system python prefix (/usr in all tested cases). This is the case regardless of whether the --system-site-packages option is used or not.

This is a continuation of the work done in ros2/build_farmer#266, #385, and #399.

Some possible solutions

1. We could stop using virtual environments in our CI entirely but this issue will also come up if any ROS users try to build packages in a workspace using virtual environments as well so without documentation it's not a total solution.

2. We could make a case upstream that there should either be a way to opt-in to installation directory options in a virtual environment and provide patches to that end.

3. We could enhance our build tools (i.e. colcon) to work around this by reading and re-providing these options via CLI where they are honored regardless.

4. We could find some way to trick or force sys.base_prefix to match sys.prefix in new virtual environments, though I don't know what other behavior would be affected by such a change going forward.

[ivanpauno]

what is the rationale to install executables in $base/lib and not in $base/bin?

If we want to still install executables in $base/lib, I think that options 2 or 3 are the best solutions.

[dirk-thomas]

what is the rationale to install executables in $base/lib and not in $base/bin?

Excutables which should not be on the global path (bin) should be placed into lib/<pkgname to be ros2 run-able.

[sloretz]

what is the rationale to install executables in $base/lib and not in $base/bin?

@ivanpauno I think our desire comes from wanting to allow ROS packages to not worry about name collisions of their binaries with other ROS packages. That's why it's $base/lib/<my pkg>/my_executable

Excutables which should not be on the global path (bin) should be placed into lib/<pkgname to be ros2 run-able.

I remember this was justified by saying the binaries are not meant to be executed by the user directly (which they're not, they're executed by ros2 run), so they can be put into $base/lib/<pkg name> because that reason is acceptable in the FSH. Re-reading it, it looks like we got it a little bit wrong. $base/lib/<pkg> is incorrect. $base/usr/lib/<pkg> would be ok, but best of all we should actually be putting them into $base/usr/libexec/<pkg>.

2. We could make a case upstream that there should either be a way to opt-in to installation directory options in a virtual environment and provide patches to that end.

My reason for bringing the FSH up, can we make the case to upstream that we need the install options in a virtualenv to comply with the FSH for executables not meant to be executed by user's directly? I wonder if that justification would carry more weight than our own reasoning of wanting to avoid name collisions.

[cottsay]

I've attempted possible solution (3) in this PR: colcon/colcon-core#530

[man-martin]

Is this still a work in progress? We used the old virtualenv version as a workaround as well but we are migrating to Debian 12 bookworm. Virtualenv 16.7.9 does not create a valid virtual environment anymore on this debian version.

Is there another workaround? The current version of colcon installed via pip in a virtual environment still ignores setup.cfg when building humble from source.