running with -D triggers debugger for skipped tests

[perrinjerome]

BUG/PROBLEM REPORT / FEATURE REQUEST

What I did:

Run tests for a package with skipped tests, like this tests/test_m.py:

import unittest

class Test1(unittest.TestCase):

    @unittest.skip('skipped !')
    def test(self):
        self.fail('1')


class Test2(unittest.TestCase):

    def test(self):
        self.assertEqual(1, 2)

with

zope-testrunner -D --path . --test-path . 

What I expect to happen:

Test1 is skipped, Test2 fails and the post mortem debugging starts.

What actually happened:

The post mortem debugger started on the exception raised internally by unittest to skip the test:

Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.


Error in test test_skipped (tests.test_m.Test1)
Traceback (most recent call last):
  File "/opt/slapgrid/d02ebd529f47acc392f6b079c09d3e4b/parts/python3/lib/python3.8/unittest/case.py", line 741, in debug
    getattr(self, self._testMethodName)()
  File "/opt/slapgrid/d02ebd529f47acc392f6b079c09d3e4b/parts/python3/lib/python3.8/unittest/case.py", line 120, in skip_wrapper
    raise SkipTest(reason)
unittest.case.SkipTest: skipped !

unittest.case.SkipTest: skipped !

> /opt/slapgrid/d02ebd529f47acc392f6b079c09d3e4b/parts/python3/lib/python3.8/unittest/case.py(120)skip_wrapper()
-> raise SkipTest(reason)
(Pdb) q
Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.

What version of Python and Zope/Addons I am using:

zope.testrunner master branch

[d-maurer]

Jérome Perrin wrote at 2022-12-6 06:27 -0800:

...

This is a Python bug. It should have been fixed in new Python versions (3.10+).

[perrinjerome]

Thanks for feedback. I am not sure we are talking about the same bug because if I try to run the tests from #142 without the fix, they fail on python 3.11 :

https://github.com/perrinjerome/zope.testrunner/actions/runs/3637480553/jobs/6138541230

Failure in test /home/runner/work/zope.testrunner/zope.testrunner/src/zope/testrunner/tests/testrunner-debugging.rst
Failed doctest test for testrunner-debugging.rst
  File "/home/runner/work/zope.testrunner/zope.testrunner/src/zope/testrunner/tests/testrunner-debugging.rst", line 0

----------------------------------------------------------------------
File "/home/runner/work/zope.testrunner/zope.testrunner/src/zope/testrunner/tests/testrunner-debugging.rst", line 121, in testrunner-debugging.rst
Failed example:
    try: testrunner.run_internal(defaults)
    finally: sys.stdin = real_stdin
    # doctest: +NORMALIZE_WHITESPACE +REPORT_NDIFF +ELLIPSIS
Differences (ndiff with -expected +actual):
      Running zope.testrunner.layer.UnitTests tests:
        Set up zope.testrunner.layer.UnitTests in N.NNN seconds.
    -   Ran 1 tests with 0 failures, 0 errors and 1 skipped in N.NNN seconds.
    + <BLANKLINE>
    + <BLANKLINE>
    + Error in test test_skipped (sample3.sampletests_d.TestSomething.test_skipped)
    + Traceback (most recent call last):
    + unittest.case.SkipTest: skipped test
    + <BLANKLINE>
    + unittest.case.SkipTest: skipped test
    + <BLANKLINE>
    + > /opt/hostedtoolcache/Python/3.11.0/x64/lib/python3.11/unittest/case.py(NNN)debug()
    + -> raise SkipTest(skip_why)
    + (Pdb) q
      Tearing down left over layers:
        Tear down zope.testrunner.layer.UnitTests in N.NNN seconds.
      False

We can see that a skipped test triggers the debugger.

Let me know if you have objections against #142 thanks !