Support changes to unittest in Python 3.11

[juliangilbey]

In Python 3.11, the format of unittest's output has changed. Whereas in Python 3.10 and earlier, the output looks like this (the tests are taken from the test suite in this package):

test_fail (testing.test_unittest.MyTest) ... FAIL
test_ok (testing.test_unittest.MyTest) ... ok

======================================================================
FAIL: test_fail (testing.test_unittest.MyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<<DIRECTORY>>/testing/test_unittest.py", line 4, in test_fail
    def test_fail(self): self.assertEqual(1+1, 3)
AssertionError: 2 != 3

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (failures=1)

in Python 3.11 (and presumably Python 3.12 and beyond), the test function name is appended to the class name, giving this output:

test_fail (testing.test_unittest.MyTest.test_fail) ... FAIL
test_ok (testing.test_unittest.MyTest.test_ok) ... ok

======================================================================
FAIL: test_fail (testing.test_unittest.MyTest.test_fail)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<<DIRECTORY>>/testing/test_unittest.py", line 4, in test_fail
    def test_fail(self): self.assertEqual(1+1, 3)
                         ^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 2 != 3

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (failures=1)

This change means that spyder shows output such as testing.test_unittest.MyTest.test_fail.test_fail when using this plugin with Python 3.11. (I know that Spyder does not claim to support 3.11 yet; indeed, debugpy is not at all ready for 3.11, but Debian is testing 3.11, so I'm fixing this issue at this point.)

This PR fixes the issue. It refactors spyder_unittest/backend/unittestrunner.py slightly, by moving the calculation of the test's full name from load_data into and new function, _get_fullname (whose behaviour depends on the version of Python being used) and calling that from try_parse_result and try_parse_exception_block.

The unit tests for unittestrunner.py also needed to be updated to handle the different behaviours, as well as the refactoring changing the behaviour of try_parse_result and try_parse_exception_block.

This PR has been tested with both Python 3.10 (old unittest behaviour) and Python 3.11 (new unittest behaviour).

[jitseniesen]

Thanks for the patch and the clear explanation.

While Spyder can't be installed in a Python 3.11 environment (unless you force it to), it can run programs in a different environment than it is installed in, including Python 3.11. So we do need to be able to parse Python 3.11 unittest output correctly. I tested your changes by hand and it does work as advertised. I did not run the automatic tests under Python 3.11, but it all looks good and I trust you.

The PR checks the Python version that Spyder is installed, which is not necessarily the same as the Python version that the tests are run under. So it does not actually work in the situation I mentioned above (Spyder installed in Python 3.10 but tests are run in Python 3.11). I'm happy to leave that to later.

The GitHub checks failed. I'm pretty sure that that has nothing to do with this PR, so I'll fix that first and I plan to then merge this PR.

[juliangilbey]

Thanks @jitseniesen! I see the problem with the version check; I hadn't considered that at all. So should I change

IS_PY311_OR_GREATER = sys.version_info[:2] >= (3, 11)

to something more like the following (though spyder/plugins/ipythonconsole/utils/kernelspec.py has a much more complicated version; spyder_unittest/widgets/unittestgui.py uses this simple scheme for determining the executable); there might well be a nicer way to do this, though!

from spyder.api.config.mixins import SpyderConfigurationAccessor
from spyder.utils import programs

def get_interpreter_version():
    config_accessor = SpyderConfigurationAccessor()
    executable = config_accessor.get_conf('executable', section='main_interpreter')

    args = ["-c", "import sys; print('.'.join(sys.version_info[:2]))"]
    proc = programs.run_program(executable, args, env={})
    exec_version = int(proc.communicate()[0])
    exec_major, exec_minor = exec_version.split(".")

    return int(exec_major), int(exec_minor)


IS_PY311_OR_GREATER = get_interpreter_version() >= (3, 11)

(I could do this via a new PR)

[jitseniesen]

That looks like a possibility, and quite easy to boot. I did not know about the SpyderConfigurationAccessor, that is a good bit to know.

If I remember correctly, the Python version is stored as a string in the dependencies member variable of the UnitTestWidget class, in dependencies['unittest']['version'] to be precise. But that information needs to be available to the UnitTestRunner to parse the unittest output correctly. Your suggestion looks simpler, at the cost of starting a new process.

If you want to do something along the lines that you suggested, or using the information already stored, please go ahead. But there is a chance that it will be overwritten by some other changes I am planning to make (though to be honest, those plans are several years old, so who know when it will happen). If your main motivation is to get the tests to pass on Debian, then feel free to stop here.

[juliangilbey]

Interesting; I hadn't noticed that. I had thought of using self.executable to determine it, but the problem I ran into was that unittestrunner.py does not get handed this UnitTestWidget object, so cannot make any use of it. (And that is good practice; the runner does not need to know about the enclosing widget.)

But a nicer way of doing this is as follows (I think). It is only the finished() method in UnittestRunner that needs to know which version of Python was used, and start() is called with the executable before finished() is called. So we could save the value of executable passed to start() in the RunnerBase object and then access it in the finished() method of UnittestRunner, perhaps using sys.executable as a fallback if something goes wrong. That ensures that exactly the same executable is used to determine the Python version as that used to run the tests.

BTW, my main motivation is to get the code to do the right thing on a Python 3.11 system; since people may well set Spyder to use a Python 3.10 virtual environment, the current version of the patch is clearly deficient!

I don't have time right now to write and test this idea, but I'll try to do so over the next few days.

[jitseniesen]

That sounds good, go ahead!

I said that "there is a chance that [your work] will be overwritten by some other changes I am planning to make", but even optimistically, my changes will only land in July or so.