Fix pytest-cov version detection with pytest >=7.0.0

[christophebedard]

I noticed that pytest-cov wasn't detected properly, even if python3-pytest-cov is installed. See for example test_tracetools here: https://build.ros2.org/job/Rpr__ros2_tracing__ubuntu_jammy_amd64/63/consoleFull#console-section-56

17:37:13 CMake Warning at /opt/ros/rolling/share/ament_cmake_pytest/cmake/ament_add_pytest_test.cmake:128 (message):
17:37:13   The Python module 'pytest-cov' was not found, test coverage will not be
17:37:13   produced.  On Linux, install the 'python3-pytest-cov' package.  On other
17:37:13   platforms, install 'pytest-cov' using pip.
17:37:13 Call Stack (most recent call first):
17:37:13   CMakeLists.txt:149 (ament_add_pytest_test)

I tested it locally and I got the same warning, but I know this used to work just fine.

Turns out that, while the plugin information printed by pytest used to be printed to stderr (which was a bit weird, I admit), it's now printed to stdout since pytest version 7.0.0: pytest-dev/pytest#8247. See the difference between pytest 7.2.2 (from pip) and pytest 6.2.5 (from system packages):

$ python3 -m pytest --version --version
This is pytest version 7.2.2, imported from /home/chris/.local/lib/python3.10/site-packages/pytest/__init__.py
setuptools registered plugins:
  pytest-cov-4.0.0 at /home/chris/.local/lib/python3.10/site-packages/pytest_cov/plugin.py
  pytest-repeat-0.9.1 at /usr/lib/python3/dist-packages/pytest_repeat.py
  pytest-timeout-2.1.0 at /usr/lib/python3/dist-packages/pytest_timeout.py
  pytest-mock-3.6.1 at /usr/lib/python3/dist-packages/pytest_mock/__init__.py
  colcon-core-0.12.1 at /usr/lib/python3/dist-packages/colcon_core/pytest/hooks.py
  pytest-rerunfailures-10.2 at /usr/lib/python3/dist-packages/pytest_rerunfailures.py
$ sudo python3 -m pytest --version --version
This is pytest version 6.2.5, imported from /usr/lib/python3/dist-packages/pytest/__init__.py
setuptools registered plugins:
  pytest-repeat-0.9.1 at /usr/lib/python3/dist-packages/pytest_repeat.py
  pytest-timeout-2.1.0 at /usr/lib/python3/dist-packages/pytest_timeout.py
  pytest-mock-3.6.1 at /usr/lib/python3/dist-packages/pytest_mock/__init__.py
  colcon-core-0.12.1 at /usr/lib/python3/dist-packages/colcon_core/pytest/hooks.py
  pytest-rerunfailures-10.2 at /usr/lib/python3/dist-packages/pytest_rerunfailures.py
  pytest-cov-3.0.0 at /usr/lib/python3/dist-packages/pytest_cov/plugin.py
$ python3 -m pytest --version --version > /dev/null
$ sudo python3 -m pytest --version --version > /dev/null
This is pytest version 6.2.5, imported from /usr/lib/python3/dist-packages/pytest/__init__.py
setuptools registered plugins:
  pytest-repeat-0.9.1 at /usr/lib/python3/dist-packages/pytest_repeat.py
  pytest-timeout-2.1.0 at /usr/lib/python3/dist-packages/pytest_timeout.py
  pytest-mock-3.6.1 at /usr/lib/python3/dist-packages/pytest_mock/__init__.py
  colcon-core-0.12.1 at /usr/lib/python3/dist-packages/colcon_core/pytest/hooks.py
  pytest-rerunfailures-10.2 at /usr/lib/python3/dist-packages/pytest_rerunfailures.py
  pytest-cov-3.0.0 at /usr/lib/python3/dist-packages/pytest_cov/plugin.py

Therefore, just search for the pytest-cov version in both stdout and stderr. This works (i.e., gets rid of the warning) locally.

[clalancette]

CI:

• Linux
• Linux-aarch64
• Windows

[christophebedard]

CI:

Note that, while this checks for build failures, it will not validate this fix. ament_add_pytest_test() only looks for pytest-cov if:

1. The AMENT_CMAKE_PYTEST_WITH_COVERAGE CMake option is enabled, and I assume it isn't enabled in normal CI runs; or
2. The package that calls ament_add_pytest_test() declares a <test_depend>python3-pytest-cov</test_depend>.
   • test_tracetools is the only core package that does this, but it only runs its tracing-related Python tests (i.e., only calls ament_add_pytest_test()) if LTTng is installed, which it isn't, currently

However, it did work locally for me, so perhaps that's enough, functionality-wise.

Ideally, a test would've caught this regression (because it is one), but this isn't really straightforward to test.

[clalancette]

1. The AMENT_CMAKE_PYTEST_WITH_COVERAGE CMake option is enabled, and I assume it isn't enabled in normal CI runs; or

I'll note that you can specify arbitrary options to those CI jobs, so you could enable it as a colcon --cmake-args option. It might be worthwhile to run another job with that enabled.

[christophebedard]

I'll note that you can specify arbitrary options to those CI jobs, so you could enable it as a colcon --cmake-args option. It might be worthwhile to run another job with that enabled.

Here:

• Linux

Here's one without this branch to make sure that the issue is indeed present:

• Linux

[christophebedard]

Looks good:

• Without this branch, there are warnings when building rclpy and many other packages which use ament_add_pytest_test() with -DAMENT_CMAKE_PYTEST_WITH_COVERAGE=ON: https://ci.ros2.org/job/ci_linux/18310/consoleFull#console-section-428
• With this branch, there are no such warnings: https://ci.ros2.org/job/ci_linux/18309/consoleFull#console-section-445

[haudren-woven]

@clalancette : I just hit this particular issue on Humble, is there any chance to backport this?

[christophebedard]

This should be backportable to Humble.

Let's see if the bot will let me do it:

[christophebedard]

@Mergifyio backport humble

[mergify]

backport humble

❌ Command disallowed due to command restrictions in the Mergify configuration.

• sender-permission>=write

[christophebedard]

No :(

@haudren-woven you can always just cherry-pick this onto humble and open a PR.

[clalancette]

@Mergifyio backport humble

[mergify]

backport humble

✅ Backports have been created

• #459 Fix pytest-cov version detection with pytest >=7.0.0 (backport #436) has been created for branch humble