gh-115421: Test that our Makefile has all needed test folders #115813

sobolevn · 2024-02-22T09:46:05Z

This PR is the alternative to #115511

While we would still have to keep adding directories to TESTSUBDIRS manually, we would now have a test to catch any cases where people forget to update / remove these entries.

This test works with --disable-test-modules as well.

Which approach do you like best?

Issue: Not all tests are installed. #115421

sobolevn · 2024-02-22T12:34:12Z

I've pushed a demo commit: f048fb5

It is missing test/archivetestdata from Makefile, some tests should fail. Let's see!

Lib/test/test_tools/test_makefile.py

sobolevn · 2024-02-22T13:44:48Z

Correctly failed jobs (which detected missing test/archivetestdata in Makefile):

Macos 14: https://github.com/python/cpython/actions/runs/8004529977/job/21862119274?pr=115813
Macos 13: https://github.com/python/cpython/actions/runs/8004529977/job/21862119510?pr=115813
Macos 14 (free-threaded): https://github.com/python/cpython/actions/runs/8004529977/job/21862119752?pr=115813
Macos 13 (free-threaded): https://github.com/python/cpython/actions/runs/8004529977/job/21862119980?pr=115813
Address Sanitizer (Ubuntu): https://github.com/python/cpython/actions/runs/8004529977/job/21862116610?pr=115813

Now, I will revert test/archivetestdata change and run all buildbots.

bedevere-bot · 2024-02-22T14:18:57Z

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit 182822d 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

sobolevn · 2024-02-22T14:31:54Z

Related failure:

======================================================================
FAIL: test_makefile_test_folders (test.test_tools.test_makefile.TestMakefile.test_makefile_test_folders)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/buildbot/buildarea/pull_request.ware-freebsd/build/Lib/test/test_tools/test_makefile.py", line 79, in test_makefile_test_folders
    self.check_existing_test_modules(actual_result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/buildbot/buildarea/pull_request.ware-freebsd/build/Lib/test/test_tools/test_makefile.py", line 61, in check_existing_test_modules
    self.assertEqual(
    ~~~~~~~~~~~~~~~~^
        unique_test_dirs.symmetric_difference(set(used)),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        set(),
        ^^^^^^
    )
    ^
AssertionError: Items in the first set but not the second:
''

bedevere-bot · 2024-02-25T08:24:42Z

🤖 New build scheduled with the buildbot fleet by @sobolevn for commit 001c14b 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

sobolevn · 2024-02-25T13:13:22Z

@Yhg1s it is ready :)

encukou

LGTM, with two suggestions:

Lib/test/test_tools/test_makefile.py

Co-authored-by: Petr Viktorin <encukou@gmail.com>

bedevere-bot · 2024-03-06T13:36:51Z

🤖 New build scheduled with the buildbot fleet by @encukou for commit 6caa8f9 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

sobolevn · 2024-03-07T11:31:48Z

Thanks everyone!

…ythonGH-115813)

bedevere-app · 2024-03-07T11:50:30Z

GH-116462 is a backport of this pull request to the 3.12 branch.

bedevere-app · 2024-03-07T11:51:06Z

GH-116462 is a backport of this pull request to the 3.12 branch.

…H-115813) (#116462) * gh-115421: Test that our Makefile has all needed test folders (GH-115813) * Update the list of installed test subdirectories --------- Co-authored-by: Nikita Sobolev <mail@sobolevn.me>

…hem (pythonGH-115813) (pythonGH-116462) * pythongh-115421: Test that our Makefile has all needed test folders (pythonGH-115813) * Update the list of installed test subdirectories --------- Co-authored-by: Nikita Sobolev <mail@sobolevn.me>

bedevere-app · 2024-03-08T12:56:11Z

GH-116498 is a backport of this pull request to the 3.11 branch.

…H-115813) This backports: - GH-115813 - GH-115422 Unlike on the main branch, new directories are added to the end, so they're a bit easier to patch out if a redistributor needs to do so. On main & 3.12, there's a special case for `idlelib/idle_test`; on 3.11 TESTSUBDIRS has several more entries that are not in `test/`. This backport ignores all of them (including idlelib). (The alternative would be list them, as additions to TEST_HOME_DIR. But that's probably too invasive; people might split stdlib up in surprising ways.) Co-authored-by: Nikita Sobolev <mail@sobolevn.me>

…ythonGH-115813)

sobolevn requested review from vstinner and Yhg1s February 22, 2024 09:46

bedevere-app bot added the awaiting core review label Feb 22, 2024

bedevere-app bot mentioned this pull request Feb 22, 2024

Not all tests are installed. #115421

Closed

sobolevn added the skip news label Feb 22, 2024

sobolevn force-pushed the issue-115421-2 branch from d9f6baf to 63ca14b Compare February 22, 2024 09:57

sobolevn mentioned this pull request Feb 22, 2024

gh-115421: List all test subdirs in Makefile by makesetup #115511

Closed

pythongh-115421: Test that our Makefile has all needed test folders

b04014f

sobolevn force-pushed the issue-115421-2 branch from 63ca14b to b04014f Compare February 22, 2024 10:09

sobolevn added 2 commits February 22, 2024 13:47

Be silent

5393647

Test PR that a missing TESTSUBDIR will fail

f048fb5

AlexWaygood reviewed Feb 22, 2024

View reviewed changes

Lib/test/test_tools/test_makefile.py Show resolved Hide resolved

Lib/test/test_tools/test_makefile.py Outdated Show resolved Hide resolved

sobolevn added 2 commits February 22, 2024 16:59

Revert test/archivetestdata change

f6c8639

Better wording

182822d

sobolevn added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 22, 2024

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 22, 2024

sobolevn added 4 commits February 22, 2024 20:13

Fix buildbot failures

d192fbe

Merge branch 'main' into issue-115421-2

88e0f0c

Do not run make, read file by lines

a743ffe

Do not touch makefile at all

001c14b

sobolevn added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 25, 2024

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Feb 25, 2024

encukou approved these changes Mar 4, 2024

View reviewed changes

Lib/test/test_tools/test_makefile.py Outdated Show resolved Hide resolved

Lib/test/test_tools/test_makefile.py Outdated Show resolved Hide resolved

bedevere-app bot removed the awaiting core review label Mar 4, 2024

bedevere-app bot added the awaiting merge label Mar 4, 2024

Apply suggestions from code review

6caa8f9

Co-authored-by: Petr Viktorin <encukou@gmail.com>

encukou added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 6, 2024

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 6, 2024

encukou merged commit 72dbea2 into python:main Mar 7, 2024
111 of 113 checks passed

bedevere-app bot removed the awaiting merge label Mar 7, 2024

encukou pushed a commit to encukou/cpython that referenced this pull request Mar 7, 2024

pythongh-115421: Test that our Makefile has all needed test folders (p…

f37cd4b

…ythonGH-115813)

encukou pushed a commit to encukou/cpython that referenced this pull request Mar 7, 2024

pythongh-115421: Test that our Makefile has all needed test folders (p…

3b51b14

…ythonGH-115813)

encukou mentioned this pull request Mar 7, 2024

[3.12] gh-115421: List all test subdirs in Makefile, and test them (GH-115813) #116462

Merged

encukou mentioned this pull request Mar 8, 2024

[3.11] gh-115421: List all test/ subdirs in Makefile, and test them (GH-115813) #116498

Merged

adorilson pushed a commit to adorilson/cpython that referenced this pull request Mar 25, 2024

pythongh-115421: Test that our Makefile has all needed test folders (p…

6ac9f0b

…ythonGH-115813)

diegorusso pushed a commit to diegorusso/cpython that referenced this pull request Apr 17, 2024

pythongh-115421: Test that our Makefile has all needed test folders (p…

0221b8d

…ythonGH-115813)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-115421: Test that our Makefile has all needed test folders #115813

gh-115421: Test that our Makefile has all needed test folders #115813

sobolevn commented Feb 22, 2024 •

edited

sobolevn commented Feb 22, 2024

sobolevn commented Feb 22, 2024

bedevere-bot commented Feb 22, 2024

sobolevn commented Feb 22, 2024

bedevere-bot commented Feb 25, 2024

sobolevn commented Feb 25, 2024

encukou left a comment

bedevere-bot commented Mar 6, 2024

sobolevn commented Mar 7, 2024

bedevere-app bot commented Mar 7, 2024

bedevere-app bot commented Mar 7, 2024

bedevere-app bot commented Mar 8, 2024

gh-115421: Test that our Makefile has all needed test folders #115813

gh-115421: Test that our Makefile has all needed test folders #115813

Conversation

sobolevn commented Feb 22, 2024 • edited

sobolevn commented Feb 22, 2024

sobolevn commented Feb 22, 2024

bedevere-bot commented Feb 22, 2024

sobolevn commented Feb 22, 2024

bedevere-bot commented Feb 25, 2024

sobolevn commented Feb 25, 2024

encukou left a comment

Choose a reason for hiding this comment

bedevere-bot commented Mar 6, 2024

sobolevn commented Mar 7, 2024

bedevere-app bot commented Mar 7, 2024

bedevere-app bot commented Mar 7, 2024

bedevere-app bot commented Mar 8, 2024

sobolevn commented Feb 22, 2024 •

edited