Removing testsrunner from testing suite

[mauzey1]

I have recently encountered an issue with testsrunner when testing the CMIP6 CV Python tests. I built a conda environment with just testsrunner and Python 3.9 to see if it was just a testsrunner issue. I tried importing the TestRunnerBase module and got the following error.

>>> from testsrunner import TestRunnerBase
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/testsrunner-v8.2_26_g3cf5a0b-py3.6.egg/testsrunner/__init__.py", line 2, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/testsrunner-v8.2_26_g3cf5a0b-py3.6.egg/testsrunner/TestRunnerBase.py", line 14, in <module>
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/testsrunner-v8.2_26_g3cf5a0b-py3.6.egg/testsrunner/image_compare.py", line 6, in <module>
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1201, in resource_filename
    return get_provider(package_or_requirement).get_resource_filename(
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 386, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 683, in find
    if dist is not None and dist not in req:
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3126, in __contains__
    return self.specifier.contains(item, prereleases=True)
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/specifiers.py", line 902, in contains
    item = Version(item)
  File "/Users/mauzey1/opt/anaconda3/envs/testsrunner_env/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py", line 197, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: 'v8.2-26-g3cf5a0b'

I'm not sure how to how fix this issue but I think this is more than enough reason to phase out the use of testsrunner from CMOR's test suite. It hasn't been updated for more than 2 years, and the HTML reports that it generates aren't really important for the CI tests.

We can run the CMIP6 CV tests using the unittest module by running the command python -m unittest Test/test_python_CMIP6_CV*.py. However, this has the issue of not working correctly with the test test_python_CMIP6_CV_terminate_signal.py. The problem is caused by the fact that these tests don't get executed as separate programs but rather as separate class objects in one Python environment. The terminate signal in CMOR is set by default as -999 but is set to signal.SIGTERM by cmor.setup(). Due to this terminate signal being held as a global variable in CMOR, it is set by the test that last used cmor.setup() or cmor.set_terminate_signal(...). With the terminate signal test executing after other tests, it always fails. I could find a way to replicate the process of testsrunner running tests as separate programs. I could also just remove the assert for the signal's value before cmor.setup() in the terminate signal test; the value of it before calling cmor.setup() might not be important.

cmor/Test/test_python_CMIP6_CV_terminate_signal.py

Lines 28 to 31 in ea1df0a

	def testTerminateSignal(self):
	self.assertEqual(cmor.get_terminate_signal(), -999)
	cmor.setup()
	self.assertEqual(cmor.get_terminate_signal(), signal.SIGTERM)