Investigate tests that are not run but should

[graingert]

There seems to be quite a few classes like this:

$ git grep 'class .*Tests:' Lib/test/ | wc -l
94

Originally posted by @Akuli in #95010 (comment)

[graingert]

@akulakov might be interested in this?

[bityob]

Hey,

I wrote a script to check all those test classes that are suspected to not run because they don't inherit from unittest.TestCase.

Shortly,

1. The script check for each such class if it is using TestCase somehow (unittest.TestCase/util.test_both/test_utils.TestCase/BaseTestCase/create_executor_tests)

2. This logic is done for the class and all it's descendants too

search-for-unused-test-classes.py

"""
Name: search-for-unused-test-classes.py
Author: @bityob

Get all unused test classes on CPython repo

Usage:
	Run this script from cpython cloned repository folder
"""

import subprocess
import re


def r(cmd, silent=False):
	output = subprocess.check_output(cmd, shell=True).decode("utf8")
	if not silent:
		print(output)
	return output


def is_class_used_by_test_case_class(curr_cls):
	try:
		# BaseTestCase, search for -> class BaseTestCase(unittest.TestCase):

		# grep -A 1, search for ->
		# 1)
		# Lib/test/test_pickle.py:class PyPersPicklerTests(AbstractPersistentPicklerTests,
		# Lib/test/test_pickle.py-                         PersistentPicklerUnpicklerMixin, unittest.TestCase):
		# 2)
		# @add_standard_options(StandardOptionsTests)
		# class ToplevelTest(AbstractToplevelTest, unittest.TestCase):

		# grep -B 1, search for ->
		# class BytesMemorySliceTest(unittest.TestCase,
    	#     BaseMemorySliceTests, BaseBytesMemoryTests):
		r(f"git grep -A 1 -B 1 {curr_cls} Lib/test/ | grep 'unittest.TestCase\\|util.test_both\\|test_utils.TestCase\\|BaseTestCase\\|create_executor_tests'", silent=True)
		return True
	except subprocess.CalledProcessError:
		return False


def get_child_classes(curr_cls, classes):
	"""
	Get all class descendants

	>>> cc = []
	>>> get_child_classes("VeryGoodTestClass", cc)
	>>> print(cc)
	["VeryGoodTestClass", VeryGoodChildTestClass"]
	"""
	try:
		responses = r(f"git grep class.*\\(.*{curr_cls}.* Lib/test/", silent=True).splitlines()

		new_classes = []
		for x in responses:
			new_cls = re.findall(f"class ([a-zA-Z0-9]+)\\(.*{curr_cls}", x.split(":")[1])[0]
			if new_cls in classes:
				continue
			new_classes.append(new_cls)

		if not new_classes:
			return []

		classes.extend(new_classes)

		for child_curr_cls in new_classes:
			classes = get_child_classes(child_curr_cls, classes)
		return classes
	except subprocess.CalledProcessError:
		return []


def search_in_child_class_for_test_case(curr_cls, all_classes=None):
	if is_class_used_by_test_case_class(curr_cls):
		return True

	all_classes = all_classes or [curr_cls]

	get_child_classes(curr_cls, all_classes)

	if len(all_classes) > 1:
		for x in all_classes[1:]:
			if is_class_used_by_test_case_class(x):
				return True

	return False


def get_class_occurences(curr_cls):
	"""
	For debugging specific class
	"""
	try:
		return r(f"git grep {curr_cls} Lib/test/", silent=False)
	except subprocess.CalledProcessError:
		return False


if __name__ == "__main__":
	classes = [x.split(":")[1][len("class "):] for x in r("git grep 'class .*Tests:' Lib/test/", silent=True).splitlines()]

	for curr_cls in classes:
		if not search_in_child_class_for_test_case(curr_cls):
			print("########")
			print(f"Need to check {curr_cls}!")

			print(f"All {curr_cls} occurences ->")
			get_class_occurences(curr_cls)

			print(f"All {curr_cls} class descendants ->")
			cc = [] 
			get_child_classes(curr_cls, cc)
			print(cc)
			print("########")

Running the script returns no results, meaning all classes were found as using unittest.TestCase somehow,
so this issue can be closed.

@iritkatriel

[iritkatriel]

Thank you @bityob. I added a bad class and confirmed that your script would have picked it up.