session.testscollected misnomer

[schettino72]

pytest version: 7.0.0

In hookspec.py

 5. Set ``session.testscollected`` to the number of collected items

But this variable actually contains the number of SELECTED tests.
The value is set at after pytest_collection_modifyitems, where items might be deselected.

To avoid breaking existing code I would suggest just deprecating this variable and adding:

• tests_selected
• tests_collected

[bluetech]

Hi @schettino72,

technically pytest_collection_modifyitems modifies session.items so there is no difference at this point between "collected items" and "selected items". Although I agree it would be good to clarify this - maybe e.g.

    4. Set ``session.items`` to the list of collected items (not including deselected items)
    5. Set ``session.testscollected`` to the number of collected items (not including deselected items)

To avoid breaking existing code I would suggest just deprecating this variable and adding:

Do you have a use case for tests_collected?

[schettino72]

technically pytest_collection_modifyitems modifies session.items so there is no difference at this point between "collected items" and "selected items".

I can not follow you...
If you look and pytest own main.py:pytest_collection_modifyitems(), it removes elements from Session.items.
So by the time self.testscollected = len(items) is called the number of collect and selected is different.

5. Set session.testscollected to the number of collected items (not including deselected items)

I guess that was the exact defintion of "selected items". 😁
I actually searched the docs, could not find a clear definition.

Output from command line usually looks like:

collected 7 items / 5 deselected / 2 selected 

On terminal.py there is a line:

        selected = self._numcollected - errors - skipped - deselected

But I guess that is a bug as well, because the errors here refer to "collect" errors, and when there is an error on collection that are zero tests collected.

There is even a guard against selected being negative (what should have been a clear indication that the computation logic is wrong).

       if self._numcollected > selected > 0:
            line += " / %d selected" % selected

Do you have a use case for tests_collected?

I found this bug while working on: numirias/pytest-json-report#74
But to tell you the truth at the moment I don't use any of these values because they are unreliable.

[schettino72]

I changed tag to BUG: if I have a single test module with 4 tests the output is:

collected 4 items

But if introduce a test module that causes an error on collection I get:

collected 4 items / 1 error / 3 selected

It should be 4 selected.

[schettino72]

I created a separate issue #9626 to handle the problem with count on terminal output.