New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytest 3.7 not collecting tests on mixed Package and Modules #3768

Closed
ZaiusDR opened this Issue Aug 2, 2018 · 5 comments

Comments

Projects
None yet
3 participants
@ZaiusDR

ZaiusDR commented Aug 2, 2018

Hi!

I've found an issue with pytest 3.7 while collecting tests on a mixed Python packages and modules folder structure, which worked correctly on previous versions of pytest. For instance if I setup a folder structure like this:

.
└── src
    ├── folder_1
    │   └── test
    │       └── test_1.py
    └── folder_2
        └── test
            └── test_2.py

Each test_*.py file containing a dummy test like:

def test_1():
    assert True

pytest collects tests correctly:

$ py.test --collect-only
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 2.7.10, pytest-3.7.0, py-1.5.4, pluggy-0.7.1
rootdir: /Users/eduardo/tests, inifile:
collected 2 items
<Module 'src/folder_1/test/test_1.py'>
  <Function 'test_1'>
<Module 'src/folder_2/test/test_2.py'>
  <Function 'test_2'>

======================================================================================= no tests ran in 0.05 seconds =======================================================================================

However if I convert one of the folders to a Python package, only the tests inside that folder are being collected:

.
└── src
    ├── __init__.py
    ├── folder_1
    │   ├── __init__.py
    │   └── test
    │       └── test_1.py
    └── folder_2
        └── test
            └── test_2.py
$ py.test --collect-only
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 2.7.10, pytest-3.7.0, py-1.5.4, pluggy-0.7.1
rootdir: /Users/eduardo/tests, inifile:
collected 1 item
<Package '/Users/eduardo/tests/src'>
  <Package '/Users/eduardo/tests/src/folder_1'>
    <Module 'test_1.py'>
      <Function 'test_1'>

======================================================================================= no tests ran in 0.03 seconds =======================================================================================

I'm not pretty sure if this is the intended behaviour or not.


Debugging info:

pip list (fresh venv tested with python2.7 and 3.6 with same results)

$ pip list
Package        Version
-------------- -------
atomicwrites   1.1.5
attrs          18.1.0
funcsigs       1.0.2
more-itertools 4.3.0
pathlib2       2.3.2
pip            18.0
pluggy         0.7.1
py             1.5.4
pytest         3.7.0
scandir        1.7
setuptools     40.0.0
six            1.11.0
wheel          0.31.1
@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 2, 2018

Member

Thanks @ZaiusDR for the report, we appreciate it.

@jonozzz

Member

nicoddemus commented Aug 2, 2018

Thanks @ZaiusDR for the report, we appreciate it.

@jonozzz

@nicoddemus

This comment has been minimized.

Show comment
Hide comment
@nicoddemus

nicoddemus Aug 2, 2018

Member

I can confirm that unfortunately this bug still exists on master (and soon to be released 3.7.1).

Member

nicoddemus commented Aug 2, 2018

I can confirm that unfortunately this bug still exists on master (and soon to be released 3.7.1).

@jonozzz

This comment has been minimized.

Show comment
Hide comment
@jonozzz

jonozzz Aug 2, 2018

I think I thought about this corner case and decided to not look at anything that's not part of a package, from inside a package. Will take a look and see if there's an easy fix.

jonozzz commented Aug 2, 2018

I think I thought about this corner case and decided to not look at anything that's not part of a package, from inside a package. Will take a look and see if there's an easy fix.

@jonozzz

This comment has been minimized.

Show comment
Hide comment
@jonozzz

jonozzz Aug 2, 2018

py.test --collect-only src would produce the expected behavior though.

jonozzz commented Aug 2, 2018

py.test --collect-only src would produce the expected behavior though.

@ZaiusDR

This comment has been minimized.

Show comment
Hide comment
@ZaiusDR

ZaiusDR Aug 6, 2018

Hey guys, would you be interested in a "minimum expected test number" feature? I mean, if something like this issue happens, it can be oversight. Maybe with a parameter to indicate a minimum number of tests ran it would be easier to notice that some of the tests haven't been executed. I could try to work on that and make a PR for you if you want.

ZaiusDR commented Aug 6, 2018

Hey guys, would you be interested in a "minimum expected test number" feature? I mean, if something like this issue happens, it can be oversight. Maybe with a parameter to indicate a minimum number of tests ran it would be easier to notice that some of the tests haven't been executed. I could try to work on that and make a PR for you if you want.

jonozzz referenced this issue Aug 10, 2018

Merge pull request #3771 from nicoddemus/package-infinite-recursion-bug
Fix infinite recursion collection bug with pytest_ignore_collect hook

jonozzz pushed a commit to jonozzz/pytest that referenced this issue Aug 10, 2018

nicoddemus added a commit that referenced this issue Aug 16, 2018

Merge pull request #3802 from jonozzz/fix-3768
Fix test collection from packages mixed with directories. #3768 and #3789

@nicoddemus nicoddemus closed this Aug 16, 2018

sfdye added a commit to pydanny/cookiecutter-django that referenced this issue Aug 19, 2018

Update pytest to 3.7.2 (#1755)
This PR updates [pytest](https://pypi.org/project/pytest) from **3.7.1** to **3.7.2**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.7.2
   ```
   =========================

Bug Fixes
---------

- `3671 &lt;https://github.com/pytest-dev/pytest/issues/3671&gt;`_: Fix ``filterwarnings`` not being registered as a builtin mark.


- `3768 &lt;https://github.com/pytest-dev/pytest/issues/3768&gt;`_, `3789 &lt;https://github.com/pytest-dev/pytest/issues/3789&gt;`_: Fix test collection from packages mixed with normal directories.


- `3771 &lt;https://github.com/pytest-dev/pytest/issues/3771&gt;`_: Fix infinite recursion during collection if a ``pytest_ignore_collect`` hook returns ``False`` instead of ``None``.


- `3774 &lt;https://github.com/pytest-dev/pytest/issues/3774&gt;`_: Fix bug where decorated fixtures would lose functionality (for example ``mock.patch``).


- `3775 &lt;https://github.com/pytest-dev/pytest/issues/3775&gt;`_: Fix bug where importing modules or other objects with prefix ``pytest_`` prefix would raise a ``PluginValidationError``.


- `3788 &lt;https://github.com/pytest-dev/pytest/issues/3788&gt;`_: Fix ``AttributeError`` during teardown of ``TestCase`` subclasses which raise an exception during ``__init__``.


- `3804 &lt;https://github.com/pytest-dev/pytest/issues/3804&gt;`_: Fix traceback reporting for exceptions with ``__cause__`` cycles.



Improved Documentation
----------------------

- `3746 &lt;https://github.com/pytest-dev/pytest/issues/3746&gt;`_: Add documentation for ``metafunc.config`` that had been mistakenly hidden.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: http://pytest.org
</details>

cmccandless added a commit to cmccandless/tools that referenced this issue Aug 20, 2018

Update pytest to 3.7.2 (#25)
This PR updates [pytest](https://pypi.org/project/pytest) from **3.7.1** to **3.7.2**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.7.2
   ```
   =========================

Bug Fixes
---------

- `3671 &lt;https://github.com/pytest-dev/pytest/issues/3671&gt;`_: Fix ``filterwarnings`` not being registered as a builtin mark.


- `3768 &lt;https://github.com/pytest-dev/pytest/issues/3768&gt;`_, `3789 &lt;https://github.com/pytest-dev/pytest/issues/3789&gt;`_: Fix test collection from packages mixed with normal directories.


- `3771 &lt;https://github.com/pytest-dev/pytest/issues/3771&gt;`_: Fix infinite recursion during collection if a ``pytest_ignore_collect`` hook returns ``False`` instead of ``None``.


- `3774 &lt;https://github.com/pytest-dev/pytest/issues/3774&gt;`_: Fix bug where decorated fixtures would lose functionality (for example ``mock.patch``).


- `3775 &lt;https://github.com/pytest-dev/pytest/issues/3775&gt;`_: Fix bug where importing modules or other objects with prefix ``pytest_`` prefix would raise a ``PluginValidationError``.


- `3788 &lt;https://github.com/pytest-dev/pytest/issues/3788&gt;`_: Fix ``AttributeError`` during teardown of ``TestCase`` subclasses which raise an exception during ``__init__``.


- `3804 &lt;https://github.com/pytest-dev/pytest/issues/3804&gt;`_: Fix traceback reporting for exceptions with ``__cause__`` cycles.



Improved Documentation
----------------------

- `3746 &lt;https://github.com/pytest-dev/pytest/issues/3746&gt;`_: Add documentation for ``metafunc.config`` that had been mistakenly hidden.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: http://pytest.org
</details>

cmccandless added a commit to cmccandless/multisite that referenced this issue Aug 20, 2018

Update pytest to 3.7.2 (#8)
This PR updates [pytest](https://pypi.org/project/pytest) from **3.6.4** to **3.7.2**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.7.2
   ```
   =========================

Bug Fixes
---------

- `3671 &lt;https://github.com/pytest-dev/pytest/issues/3671&gt;`_: Fix ``filterwarnings`` not being registered as a builtin mark.


- `3768 &lt;https://github.com/pytest-dev/pytest/issues/3768&gt;`_, `3789 &lt;https://github.com/pytest-dev/pytest/issues/3789&gt;`_: Fix test collection from packages mixed with normal directories.


- `3771 &lt;https://github.com/pytest-dev/pytest/issues/3771&gt;`_: Fix infinite recursion during collection if a ``pytest_ignore_collect`` hook returns ``False`` instead of ``None``.


- `3774 &lt;https://github.com/pytest-dev/pytest/issues/3774&gt;`_: Fix bug where decorated fixtures would lose functionality (for example ``mock.patch``).


- `3775 &lt;https://github.com/pytest-dev/pytest/issues/3775&gt;`_: Fix bug where importing modules or other objects with prefix ``pytest_`` prefix would raise a ``PluginValidationError``.


- `3788 &lt;https://github.com/pytest-dev/pytest/issues/3788&gt;`_: Fix ``AttributeError`` during teardown of ``TestCase`` subclasses which raise an exception during ``__init__``.


- `3804 &lt;https://github.com/pytest-dev/pytest/issues/3804&gt;`_: Fix traceback reporting for exceptions with ``__cause__`` cycles.



Improved Documentation
----------------------

- `3746 &lt;https://github.com/pytest-dev/pytest/issues/3746&gt;`_: Add documentation for ``metafunc.config`` that had been mistakenly hidden.
   ```
   
  
  
   ### 3.7.1
   ```
   =========================

Bug Fixes
---------

- `3473 &lt;https://github.com/pytest-dev/pytest/issues/3473&gt;`_: Raise immediately if ``approx()`` is given an expected value of a type it doesn&#39;t understand (e.g. strings, nested dicts, etc.).


- `3712 &lt;https://github.com/pytest-dev/pytest/issues/3712&gt;`_: Correctly represent the dimensions of an numpy array when calling ``repr()`` on ``approx()``.

- `3742 &lt;https://github.com/pytest-dev/pytest/issues/3742&gt;`_: Fix incompatibility with third party plugins during collection, which produced the error ``object has no attribute &#39;_collectfile&#39;``.

- `3745 &lt;https://github.com/pytest-dev/pytest/issues/3745&gt;`_: Display the absolute path if ``cache_dir`` is not relative to the ``rootdir`` instead of failing.


- `3747 &lt;https://github.com/pytest-dev/pytest/issues/3747&gt;`_: Fix compatibility problem with plugins and the warning code issued by fixture functions when they are called directly.


- `3748 &lt;https://github.com/pytest-dev/pytest/issues/3748&gt;`_: Fix infinite recursion in ``pytest.approx`` with arrays in ``numpy&lt;1.13``.


- `3757 &lt;https://github.com/pytest-dev/pytest/issues/3757&gt;`_: Pin pathlib2 to ``&gt;=2.2.0`` as we require ``__fspath__`` support.


- `3763 &lt;https://github.com/pytest-dev/pytest/issues/3763&gt;`_: Fix ``TypeError`` when the assertion message is ``bytes`` in python 3.
   ```
   
  
  
   ### 3.7.0
   ```
   =========================

Deprecations and Removals
-------------------------

- `2639 &lt;https://github.com/pytest-dev/pytest/issues/2639&gt;`_: ``pytest_namespace`` has been deprecated.

  See the documentation for ``pytest_namespace`` hook for suggestions on how to deal
  with this in plugins which use this functionality.


- `3661 &lt;https://github.com/pytest-dev/pytest/issues/3661&gt;`_: Calling a fixture function directly, as opposed to request them in a test function, now issues a ``RemovedInPytest4Warning``. It will be changed into an error in pytest ``4.0``.

  This is a great source of confusion to new users, which will often call the fixture functions and request them from test functions interchangeably, which breaks the fixture resolution model.



Features
--------

- `2283 &lt;https://github.com/pytest-dev/pytest/issues/2283&gt;`_: New ``package`` fixture scope: fixtures are finalized when the last test of a *package* finishes. This feature is considered **experimental**, so use it sparingly.


- `3576 &lt;https://github.com/pytest-dev/pytest/issues/3576&gt;`_: ``Node.add_marker`` now supports an ``append=True/False`` parameter to determine whether the mark comes last (default) or first.


- `3579 &lt;https://github.com/pytest-dev/pytest/issues/3579&gt;`_: Fixture ``caplog`` now has a ``messages`` property, providing convenient access to the format-interpolated log messages without the extra data provided by the formatter/handler.


- `3610 &lt;https://github.com/pytest-dev/pytest/issues/3610&gt;`_: New ``--trace`` option to enter the debugger at the start of a test.


- `3623 &lt;https://github.com/pytest-dev/pytest/issues/3623&gt;`_: Introduce ``pytester.copy_example`` as helper to do acceptance tests against examples from the project.



Bug Fixes
---------

- `2220 &lt;https://github.com/pytest-dev/pytest/issues/2220&gt;`_: Fix a bug where fixtures overridden by direct parameters (for example parametrization) were being instantiated even if they were not being used by a test.


- `3695 &lt;https://github.com/pytest-dev/pytest/issues/3695&gt;`_: Fix ``ApproxNumpy`` initialisation argument mixup, ``abs`` and ``rel`` tolerances were flipped causing strange comparsion results.
  Add tests to check ``abs`` and ``rel`` tolerances for ``np.array`` and test for expecting ``nan`` with ``np.array()``


- `980 &lt;https://github.com/pytest-dev/pytest/issues/980&gt;`_: Fix truncated locals output in verbose mode.



Improved Documentation
----------------------

- `3295 &lt;https://github.com/pytest-dev/pytest/issues/3295&gt;`_: Correct the usage documentation of ``--last-failed-no-failures`` by adding the missing ``--last-failed`` argument in the presented examples, because they are misleading and lead to think that the missing argument is not needed.



Trivial/Internal Changes
------------------------

- `3519 &lt;https://github.com/pytest-dev/pytest/issues/3519&gt;`_: Now a ``README.md`` file is created in ``.pytest_cache`` to make it clear why the directory exists.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: http://pytest.org
</details>

bors bot added a commit to rehandalal/therapist that referenced this issue Aug 20, 2018

Merge #28
28: Update pytest to 3.7.2 r=rehandalal a=pyup-bot


This PR updates [pytest](https://pypi.org/project/pytest) from **3.7.1** to **3.7.2**.



<details>
  <summary>Changelog</summary>
  
  
   ### 3.7.2
   ```
   =========================

Bug Fixes
---------

- `3671 &lt;https://github.com/pytest-dev/pytest/issues/3671&gt;`_: Fix ``filterwarnings`` not being registered as a builtin mark.


- `3768 &lt;https://github.com/pytest-dev/pytest/issues/3768&gt;`_, `3789 &lt;https://github.com/pytest-dev/pytest/issues/3789&gt;`_: Fix test collection from packages mixed with normal directories.


- `3771 &lt;https://github.com/pytest-dev/pytest/issues/3771&gt;`_: Fix infinite recursion during collection if a ``pytest_ignore_collect`` hook returns ``False`` instead of ``None``.


- `3774 &lt;https://github.com/pytest-dev/pytest/issues/3774&gt;`_: Fix bug where decorated fixtures would lose functionality (for example ``mock.patch``).


- `3775 &lt;https://github.com/pytest-dev/pytest/issues/3775&gt;`_: Fix bug where importing modules or other objects with prefix ``pytest_`` prefix would raise a ``PluginValidationError``.


- `3788 &lt;https://github.com/pytest-dev/pytest/issues/3788&gt;`_: Fix ``AttributeError`` during teardown of ``TestCase`` subclasses which raise an exception during ``__init__``.


- `3804 &lt;https://github.com/pytest-dev/pytest/issues/3804&gt;`_: Fix traceback reporting for exceptions with ``__cause__`` cycles.



Improved Documentation
----------------------

- `3746 &lt;https://github.com/pytest-dev/pytest/issues/3746&gt;`_: Add documentation for ``metafunc.config`` that had been mistakenly hidden.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: http://pytest.org
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment