Issues with colcon test --python-testing pytest

[methylDragon]

Hi there! I'm having issues using colcon test. The details are below.

The Issue

Calling colcon test --packages-select rmf_mock_adapter_python --python-testing pytest does not seem to work.

The issue also persists if the --python-testing setuppy_test argument is used as well.

I have a toy test in the <package-dir>/tests directory, and the tests directory has an __init__.py.
But that doesn't matter because tests don't even begin to get run.

Calling colcon test without the --python-testing argument specified causes the error messages listed below to disappear. But the package fails with an [ Exited with code 2 ]

Calling pytest manually in the tests directory shows test results as expected.

Not sure if it's an issue with my set up or with colcon.

Environment Details

Ubuntu 18.04

Colcon versions

colcon-argcomplete 0.3.3: up-to-date
colcon-bash 0.4.2: up-to-date
colcon-cd 0.1.1: up-to-date
colcon-cmake 0.2.22: up-to-date
colcon-core 0.5.9: up-to-date
colcon-defaults 0.2.5: up-to-date
colcon-devtools 0.2.2: up-to-date
colcon-library-path 0.2.1: up-to-date
colcon-metadata 0.2.4: up-to-date
colcon-notification 0.2.13: up-to-date
colcon-output 0.2.9: up-to-date
colcon-package-information 0.3.3: up-to-date
colcon-package-selection 0.2.6: up-to-date
colcon-parallel-executor 0.2.4: up-to-date
colcon-pkg-config 0.1.0: up-to-date
colcon-powershell 0.3.6: up-to-date
colcon-python-setup-py 0.2.5: up-to-date
colcon-recursive-crawl 0.2.1: up-to-date
colcon-ros 0.3.17: up-to-date
colcon-test-result 0.3.8: up-to-date
colcon-zsh 0.4.0: up-to-date

setup.py

Relevant sections. Everything else builds normally, only the test messes up.

setup(
    name=package_name,
    version=__version__,
    packages=[],
    author='methylDragon',
    author_email='methylDragon@gmail.com',
    url='',
    description='Python bindings for rmf_mock_adapter',
    long_description='',
    ext_modules=[CMakeExtension(package_name)],
    setup_requires=['pybind11>=2.5.0', 'pytest-runner'],
    cmdclass={'build_ext': CMakeBuild},
    zip_safe=False,
    entry_points={'console_scripts': []},
    license='Apache License, Version 2.0',
    install_requires=['setuptools'],
    data_files=[
        ('share/ament_index/resource_index/packages',
            ['resource/' + package_name]),
        ('share/' + package_name, ['package.xml'])
    ],
    tests_require=['pytest'],
    test_suite="tests"
)

Error messages

Starting >>> rmf_mock_adapter_python
--- stderr: rmf_mock_adapter_python                   
Traceback (most recent call last):
  File "/home/methyldragon/.local/lib/python3.6/site-packages/colcon_core/task/python/test/__init__.py", line 75, in test
    return await extension.step(self.context, env, setup_py_data)
AttributeError: 'NoneType' object has no attribute 'step'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/methyldragon/.local/lib/python3.6/site-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/home/methyldragon/.local/lib/python3.6/site-packages/colcon_core/task/__init__.py", line 92, in __call__
    return await task_method(*args, **kwargs)
  File "/home/methyldragon/.local/lib/python3.6/site-packages/colcon_ros/task/ament_python/test.py", line 29, in test
    return await extension.test()
  File "/home/methyldragon/.local/lib/python3.6/site-packages/colcon_core/task/python/test/__init__.py", line 81, in test
    "'{extension.STEP_TYPE}': {e}\n{exc}".format_map(locals()))
AttributeError: 'NoneType' object has no attribute 'STEP_TYPE'
---
Failed   <<< rmf_mock_adapter_python	[ Exited with code 1 ]

Summary: 0 packages finished [0.50s]
  1 package failed: rmf_mock_adapter_python
  1 package had stderr output: rmf_mock_adapter_python
'NoneType' object has no attribute 'STEP_TYPE'

[dirk-thomas]

Please provide a SSCCE to easily reproduce your problem.

--

The CLI help for --python-testing states:

The Python testing framework to use (default: determined based on the packages tests_require)

Since your package declares tests_require=['pytest'] you don't have to specify it - it will default to pytest. Certainly passing setuppy_test explicitly would be wrong.

[dirk-thomas]

As it turns out passing --python-testing is broken at the moment, It will be addressed by colcon/colcon-core#355.

For this specific issue not passing the argument should still work and use pytest to run the tests.

[methylDragon]

Minimal Example

pytest_minimal.zip

---

• Running python3 setup.py test in the package directory allows for the tests to work with output

~/Desktop/pybind_ws/src/pytest_minimal$ python3 setup.py test
running pytest
running egg_info
creating pytest_minimal.egg-info
writing pytest_minimal.egg-info/PKG-INFO
writing dependency_links to pytest_minimal.egg-info/dependency_links.txt
writing entry points to pytest_minimal.egg-info/entry_points.txt
writing requirements to pytest_minimal.egg-info/requires.txt
writing top-level names to pytest_minimal.egg-info/top_level.txt
writing manifest file 'pytest_minimal.egg-info/SOURCES.txt'
reading manifest file 'pytest_minimal.egg-info/SOURCES.txt'
writing manifest file 'pytest_minimal.egg-info/SOURCES.txt'
running build_ext
================================================ test session starts =================================================
platform linux -- Python 3.6.9, pytest-5.4.3, py-1.8.1, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/methyldragon/Desktop/pybind_ws/src/pytest_minimal, inifile: setup.cfg, testpaths: tests
plugins: ament-lint-0.8.1, ament-pep257-0.8.1, ament-copyright-0.8.1, ament-flake8-0.8.1, ament-xmllint-0.8.1, rerunfailures-8.0, colcon-core-0.5.9, repeat-0.8.0, cov-2.8.1
collected 1 item                                                                                                     

tests/test_minimal.py::test_minimal FAILED                                                                     [100%]

====================================================== FAILURES ======================================================
____________________________________________________ test_minimal ____________________________________________________

    def test_minimal():
        assert True
>       assert False, "OH NO"
E       AssertionError: OH NO
E       assert False

tests/test_minimal.py:3: AssertionError
============================================== short test summary info ===============================================
FAILED tests/test_minimal.py::test_minimal - AssertionError: OH NO
================================================= 1 failed in 0.05s ==================================================

Issue

• Running colcon test --packages-select pytest_minimal --pytest-args --verbose in the workspace after running colcon build gives no output:

~/Desktop/pybind_ws$ colcon test --packages-select pytest_minimal --pytest-args --verbose

Starting >>> pytest_minimal
Finished <<< pytest_minimal [0.96s]     [ with test failures ]

Summary: 1 package finished [1.22s]
  1 package had test failures: pytest_minimal

Seems the tests run, but how do I get the test output?

Thanks again for all the help!

[dirk-thomas]

python3 setup.py test

This (setup.py test) is a deprecated usage pattern. You should invoke pytest directly instead.

Seems the tests run, but how do I get the test output?

• Either you modify the test invocation to show stdout output (e.g. with --event-handlers console_direct+ when processing a single package). By default colcon only shows stderr and pytest outputs all content to stdout.
• Or you call colcon test-result [--verbose] afterwards.

[methylDragon]

Ah, alright, thanks! I'll close the issue then (: 👍