Regression tests do not support exclusion and pgo in the same invocation

[Kaelten]

Bug report

Bug description:

Background

I've been working to build from source on Alpine. Starting in 3.13 pgo optimizations now error and that led to the test_re tests failing and blocking builds. I didn't want to fully disable pgo optimizations and miss out on all of the related performance wins.

Attempted Fix and Problem Identification

I initially attempted to preclude this by leveraging PROFILE_TASK="-m test --pgo -x test_re" when calling make. This had the impact of making all tests except the pgo tests run. After some spelunking I found that this logic from Lib/test/libregrtest/main.py is the culprit.

def find_tests(self, tests: TestList | None = None) -> tuple[TestTuple, TestList | None]:
        if tests is None:
            tests = []
        if self.single_test_run:
            self.next_single_filename = os.path.join(self.tmp_dir, 'pynexttest')
            try:
                with open(self.next_single_filename, 'r') as fp:
                    next_test = fp.read().strip()
                    tests = [next_test]
            except OSError:
                pass

        if self.fromfile:
            tests = []
            # regex to match 'test_builtin' in line:
            # '0:00:00 [  4/400] test_builtin -- test_dict took 1 sec'
            regex = re.compile(r'\btest_[a-zA-Z0-9_]+\b')
            with open(os.path.join(os_helper.SAVEDCWD, self.fromfile)) as fp:
                for line in fp:
                    line = line.split('#', 1)[0]
                    line = line.strip()
                    match = regex.search(line)
                    if match is not None:
                        tests.append(match.group())

        strip_py_suffix(tests)

        if self.pgo:
            # add default PGO tests if no tests are specified
            setup_pgo_tests(self.cmdline_args, self.pgo_extended)

        if self.tsan:
            setup_tsan_tests(self.cmdline_args)

        if self.tsan_parallel:
            setup_tsan_parallel_tests(self.cmdline_args)

        exclude_tests = set()
        if self.exclude:
            for arg in self.cmdline_args:
                exclude_tests.add(arg)
            self.cmdline_args = []

        alltests = findtests(testdir=self.test_dir,
                             exclude=exclude_tests)

        if not self.fromfile:
            selected = tests or self.cmdline_args
            if selected:
                selected = split_test_packages(selected)
            else:
                selected = alltests
        else:
            selected = tests

        if self.single_test_run:
            selected = selected[:1]
            try:
                pos = alltests.index(selected[0])
                self.next_single_test = alltests[pos + 1]
            except IndexError:
                pass

        # Remove all the selected tests that precede start if it's set.
        if self.starting_test:
            try:
                del selected[:selected.index(self.starting_test)]
            except ValueError:
                print(f"Cannot find starting test: {self.starting_test}")
                sys.exit(1)

        random.seed(self.random_seed)
        if self.randomize:
            random.shuffle(selected)

        for priority_test in reversed(self.prioritize_tests):
            try:
                selected.remove(priority_test)
            except ValueError:
                print(f"warning: --prioritize={priority_test} used"
                        f" but test not actually selected")
                continue
            else:
                selected.insert(0, priority_test)

        return (tuple(selected), tests)

Due to the order of operations what ends up happening is the pgo test suite gets added to cmdline_args and then they all get excluded leaving everything else to run.

Hacky Workaround

My eventual workaround is... less than great. I ultimately settled on two choices: explicitly maintain my own copy of the PGO test list in my dockerfile or update the list defined in the testing framework. I opted for the later and have added this to my build script.

sed -Ei "s/'test_re',?$//g"  Lib/test/libregrtest/pgo.py

This felt like the less likely option to break over time as we not only build 3.13 but other versions as well.

Suggested Improvement Options

1. Automatic Exclusion for Musl:
Automatically exclude incompatible tests on musl-based systems, which appears consistent with other test behaviors. This minimizes friction for maintainers who otherwise may forgo PGO optimizations.

From my view, this is the preferred option.

2. Improved Test Exclusion Logic:
Adjust --pgo and -x flag logic to allow intuitive exclusion of PGO tests, or otherwise enable this behavior, and combine with clearer documentation for builders.

3. Explicit Error Messaging:
Explicitly error when these flags are used together to reduce confusion and accidental behaviors. This aligns with other checks for mutually exclusive flags.

CPython versions tested on:

3.13

Operating systems tested on:

Other