[CI/CD] allow to skip tests that are know to lack a C (or Python) implementation.

[picnixz]

I don't know whether it's already possible to do it but I would like to have the possibility to make the CI/CD pass (or maybe fail to avoid surprises, but with a specific job) even if I do not have yet the C implementation. The idea is to have the following possibility:

class MixinTest:
	def test(self):
		# some code that is tested with something written in C or in Python
		...

	@test.support.has_implementation('c')
	def test_c_only(self):
		# some code that is tested only if something is written in C
		...

	@test.support.has_implementation('py')
	def test_python_only(self):
		# some code that is tested only if something is written in Python
		...

In #120728, I currently have the tests skipped because whenever I push a commit, the CI/CD would try to run tests for both the Python implementation and the C implementation (e.g., for BytesIO, or datetime). This might be quite annoying when we only want to add the Python implementation and check whether the tests are correct or not (if they already fail here, it doesn't make sense to implement the C implementation).

Actually, this issue essentially arises from the fact that mixin classes are used to avoid test duplications, but those mixin classes do not give the possibility to check whether the interface being tested is the C or the Python implementation (for instance, we could have data = self.myclass(...) where myclass is _pyio.BytesIO (in Python) or io.BytesIO (in C)).

[picnixz]

After some thoughts, here is the interface that I could suggest:

• first, this special decorator should never be merged in PRs unless there's a good reason (but I don't see which reason we could have since it would only be used with mixin classes parametrized by different interfaces)

• since most of the time, everything is based according to a class attribute of the mixin class, it'd be easier to use:

  # skip if 'self.something' raises AttributeError, 
  # or if the C implementation of 'self.something' is missing. 
  @test.support.ignore_missing_impl(something="c") 
  def test_something(self):
      x = self.something(...)

I still don't know how to elegantly skip tests where a method is not implemented in C but is implemented in Python without having a complex interface. One idea is to have some generic predicate like:

def test(self):
	x = self.something_in_c_and_python(...)
	# assume that x.foo() exists only in Python
	
	# would call self.skipTest() if x.c does not exist
	# and if x is an object from a C module instead.
	# Usually the files import both the C and Python implementations,
	# so we could check whether x is from the C or the Python implementation like that.
	test.support.check_support(self, x, c='foo', pymodule=_pymodule, cmodule=_cmodule)
    y = x.foo()

[terryjreedy]

I think you should just comment out or explicitly skip tests that are currently inapplicable. As you add pieces of the C implementation, you want them tested, so uncomment/unskip that tests that should pass with the new code addition.

[picnixz]

@terryjreedy Actually, after a good sleep, I observed that a simpler solution is just to add a check inside the test like

if self.myclass is not _pyio.BytesIO:
	self.skipTest("TODO")

Ideally, I'd like to have the possibility of having self.skipIf(self.myclass is not _pyio.BytesIO) but there is only a skipIf decorator and not a skipIf method. Would it make sense to expose some skipIf method to the unittest class as well?

Anyway, I'll close this one as not planned.

[terryjreedy]

I don't see the purpose of a method. What would it skip? The decorator skips what it contains and skipTest skips what contains it. That seems to cover the needed cases pretty well.

[picnixz]

The issue with the decorator is that I cannot make it skip something that depends on an attribute, i.e., I cannot use self of something (at least, I don't see how I can put a skipIf decorator on a base class method which is then inherited by C or Python interfaces).

Maybe an example would be more explicit:

class MixinClass:
    myclass = None

    def test_this(self):
        x = self.myclass(...)

	def test_incomplete_component_for_now(self):
		# test is implementation agnostic
		if self.myclass is not MyPythonClass:
			self.skipTest("not ready for this test")

	# I would like to use `skipIf` like that:
	def test_incomplete_component_for_now_with_skipif(self):
		# self.skipIf does not exist as method on test cases!
		# and I cannot use the decorator on base mixin classes
		self.skipIf(self.myclass is not MyPythonClass)

class PythonTestSuite(MixinClass):
    myclass = MyPythonClass

class CTestSuite(MixinClass):
    myclass = MyCClass

Note that I don't want to disable the entire test suite of C classes, just specific tests on mixin classes, but I expected that skipIf would also be available as a method actually (it's more synctatic sugar but I understand if you don't want that additional interface).