Meta issue: Mark all CPython-specific tests with a test.support.cpython_only decorator

[Eclips4]

Bug report

Bug description:

There's a certain amount of Cpython-specific tests which isn't marked with a cpython_only decorator.

I'm certain that CPython-specific tests should be excluded from running on any other implementation, which is advantageous for us. This ensures that when developing another Python implementation, you can selectively avoid running these specific CPython tests, which is beneficial during the testing phase. This way, you can focus on relevant tests for your implementation without unnecessary interference from CPython-specific ones.

So, my final goal is to make the whole CPython test suite runnable on other Python implementations.

I would like to send a one PR per test file which needs a fix.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-115142: Skip test_optimizer if _testinternalcapi module is not available. #115175
• gh-115142: Skip test__xxsubinterpreters if _testinternalcapi is not available #116507
• [3.12] gh-115142: Skip test__xxsubinterpreters if _testcapi is not available #116543
• gh-115142: Skip test_capi/test_opt.py if _testinternalcapi is not available #117574
• gh-115142: Skip test_capi.test_dict.py if _testcapi and _testlimitedcapi are not available #117588
• [3.12] gh-115142: Skip test_capi.test_dict.py if _testcapi an… #117718

[serhiy-storchaka]

How are you going to determine that they are CPython-specific?

[Eclips4]

How are you going to determine that they are CPython-specific?

It is a good question.
First of all I would like to "transform" all tests which using CPython-specific modules (for example, _testinternalcapi and _elementtree)
Secondly, if test using a some private (I mean non-documented) attributes, does this mean that this test is CPython specific?

[terryjreedy]

This is a complicated issue that to me needs some meta-level discussion and possible inclusion of representatives of other implementations. I have several questions.

0. Benefits to who? There are Python developers, CPython d Quoting the initial message

I'm certain that CPython-specific tests should be excluded from running on any other implementation,

Why? The answer depends on how broadly or narrowly 'CPython-specific' is defined.

which is advantageous for us.

'Us' as Python developers, or as CPython developers? I am responding mostly as a Python developer whose C knowledge has faded somewhat.

This ensures that when developing another Python implementation, you can selectively avoid running these specific CPython tests, which is beneficial during the testing phase.

cpython-only is all or nothing for the items marked. See next question.

1. Skip level: assert, method, class, or module? I believe there was a recent patch that added perhaps a hundred decorators to a hundred methods. I would not like to read or edit such a file. If I were implementing the tested module, I doubt I would want to have to remove or comment out the hundred or so decorators one by one. I think most of the skipping should be at the class or even module level. If unittest cannot handle a class being skipped when called, as opposed to when it calls setUpModule, we should try to change it.

Perhaps implementation skips of asserts should be discouraged. There may be some OS-dependent assert in IDLE test, but that is another matter.

2. C-specific or CPython-specific? Which is true of C-coded modules like _elementtree? Is it not usable by any implementation that is either based on or can access C code? Things that are only C-specific should not be marked CPython-specific. In other words, do C-coded stdlib modules use the private C-API, however defined?

3. Python objects: Public Python objects should not be CPython specific. Some Python-visible behaviors are and when possibly depended on is a manner we don't want, are documented as such. Still, if other implementations wish to imitate the behavior, they need to easily run the tests. Same comment for Python-coded private objects.

If implementations that copy the module code also copy the test code, what is the problem? Or perhaps, should there be a problem? Do we have it as a goal that (with rare documented exceptions) Python-coded stdlib modules only depend on Python, and not CPython-specific behaviors? Should they avoid, for instance, depending on CPython's specific int caching? One way to test and catch this, would be to have an alternate test build that behaved otherwise, such as by disabling the caching.

[serhiy-storchaka]

See also #69317. PyPy uses modified CPython tests. We can look at that modifications and decide which of them are CPython-specific and which are PyPy-specific.

[Eclips4]

This is a complicated issue that to me needs some meta-level discussion and possible inclusion of representatives of other implementations. I have several questions.

0. Benefits to who? There are Python developers, CPython d Quoting the initial message

I'm certain that CPython-specific tests should be excluded from running on any other implementation,

Why? The answer depends on how broadly or narrowly 'CPython-specific' is defined.

which is advantageous for us.

'Us' as Python developers, or as CPython developers? I am responding mostly as a Python developer whose C knowledge has faded somewhat.

Sorry, I was unclear here. It's not make any benefit for CPython developers, but there are obvious benefits for Python ecosystem - testing other Python implementations will be easier.

Skip level: assert, method, class, or module? I believe there was a recent patch that added perhaps a hundred decorators to a hundred methods. I would not like to read or edit such a file. If I were implementing the tested module, I doubt I would want to have to remove or comment out the hundred or so decorators one by one. I think most of the skipping should be at the class or even module level. If unittest cannot handle a class being skipped when called, as opposed to when it calls setUpModule, we should try to change it.

That's depends on test which we'd like to transform. Tests like test_optimizer definitely should be skipped on module level.
For instance, if some method testing CPython int caching, this method should be decorated with a cpython_only.

C-specific or CPython-specific? Which is true of C-coded modules like _elementtree? Is it not usable by any implementation that is either based on or can access C code? Things that are only C-specific should not be marked CPython-specific. In other words, do C-coded stdlib modules use the private C-API, however defined?

It's a good question! I think we talking about CPython-specific tests. For example, there are tests for _elementtree module (which is actually a C-extension). Do we need to skip tests for this module on non-CPython implementations? I think no. This module is not required to be present in stdlib, but what prevents other implementation from having it? So in my opinion we must skip this test if _elementtree module is not available.

[Eclips4]

One more detail: I think these kind of changes should be backportable where it is possible.