Adding positional-only parameter notation (/ in function signature)

[Matiiss]

Closes #2445

TODO:

• lowercase names where that was previous pos only param convention
• fix the ignored names
• fix merge conflicts
• fix CI error

I'm planning on also removing type names from function signatures in the docs as that seems to be how positional-only parameters were notated before, also removing parentheses from function signatures in the docs, basically adopting positional-only parameter notation to its fullest extent.

Also haven't touched up on Python implementations and Cython (that one probably won't do at all?).

Some random notes (so I don't lose them):

regex: METH_VARARGS(?! \| METH_KEYWORDS)|METH_O|METH_FASTCALL
files to include: src_c

notes along the way:
change parameter names starting with a capital letter in function signatures in the docs to lowercase
in font.pyi and font.rst swap metrics and get_italic
in event.pyi remove None function signatures
remove types from function signatures and replace with the actual param name
joystick.rst some method names include signatures
pygame.joystick.Joystick is a function...
set_repeat in keys.rst
ignoring newbuffer.c
ignoring touch.c although it resembles C a lot
ignoring static.c

A script I used for testing/checking that all functions that are implemented in C and don't accept keyword arguments are properly annotated in the stubs and docs (it's a bit of a mess).
The excluded doc names should be investigated further, most of them will get solved by changing what I mentioned above about general signature consistency and such.

import os
import re


# excluded: locals, sprite, examples, tests, version
modules = [
    ("color", "color.c", "color.pyi", "color.rst"),
    ("display", "display.c", "display.pyi", "display.rst"),
    ("draw", "draw.c", "draw.pyi", "draw.rst"),
    ("event", "event.c", "event.pyi", "event.rst"),
    ("font", "font.c", "font.pyi", "font.rst"),
    ("image", "image.c", "image.pyi", "image.rst"),
    ("key", "key.c", "key.pyi", "key.rst"),
    ("mixer", "mixer.c", "mixer.pyi", "mixer.rst"),
    ("mouse", "mouse.c", "mouse.pyi", "mouse.rst"),
    ("rect", "rect.c", "rect.pyi", "rect.rst"),
    ("surface", "surface.c", "surface.pyi", "surface.rst"),
    ("time", "time.c", "time.pyi", "time.rst"),
    ("mixer.music", "music.c", "mixer_music.pyi", "music.rst"),
    ("pygame", "base.c", "base.pyi", "pygame.rst"),
    # ("cursors", "cursors.c", "cursors.pyi", "cursors.rst"),  # python implementation
    ("joystick", "joystick.c", "joystick.pyi", "joystick.rst"),
    ("mask", "mask.c", "mask.pyi", "mask.rst"),
    ("transform", "transform.c", "transform.pyi", "transform.rst"),
    ("bufferproxy", "bufferproxy.c", "bufferproxy.pyi", "bufferproxy.rst"),
    ("freetype", "_freetype.c", "freetype.pyi", "freetype.rst"),
    ("gfxdraw", "gfxdraw.c", "gfxdraw.pyi", "gfxdraw.rst"),
    # ("midi", "midi.c", "midi.pyi", "midi.rst"),  # python implementation
    ("pixelarray", "pixelarray.c", "pixelarray.pyi", "pixelarray.rst"),
    ("pixelcopy", "pixelcopy.c", "pixelcopy.pyi", "pixelcopy.rst"),
    # ("sndarray", "sndarray.c", "sndarray.pyi", "sndarray.rst"),  # python implementation
    # ("surfarray", "surfarray.c", "surfarray.pyi", "surfarray.rst"),  # python implementation
    ("math", "math.c", "math.pyi", "math.rst"),
    # ("camera", "camera.c", "camera.pyi", "camera.rst"),  # python implementation
    # ("controller", "controller.c", "controller.pyi", "controller.rst"),  cython implementation
    ("scrap", "scrap.c", "scrap.pyi", "scrap.rst"),
    ("system", "system.c", "system.pyi", "system.rst"),
    # ("touch", "touch.c", "touch.pyi", "touch.rst"),  # cython implementation
    ("window", "window.c", "_window.pyi", "sdl2_video.rst"),
]

# these are alright
ignore_stub_names = {
    "convert",
    "convert_alpha",
}

# these are alright too, explanation in comments
ignore_doc_names = {
    "set_blocked",  # None
    "set_allowed",  # None
    "from_polar",  # regex fail mostly
    "from_spherical",  # regex fail mostly
    "set_endevent",  # no args
    "set_endevent",  # not detected at all, but no args
    "update",  # rect.update, regex fail mostly
    "clipline",  # regex fail mostly
    "collidepoint",  # regex fail mostly
    "fblits",  # regex fail mostly
    "convert",  # no args
    "convert_alpha",  # no args
    "set_colorkey",  # None
    "set_alpha",  # None
    "get_at",  # regex fail mostly
    "set_at",  # regex fail mostly
    "get_at_mapped",  # regex fail mostly
    "set_palette",  # regex fail mostly
    "set_clip",  # None
    "set_masks",  # regex fail mostly
    "set_shifts",  # regex fail mostly
}

# these are fine the way they are
ignore_uppercase_params = {"set_palette", "set_palette_at"}

# this will be fixed in a different PR
ignore_uppercase_params_modules = {"math"}

use_specific = None

SRC_C = "src_c"
STUBS = "buildconfig/stubs/pygame"
DOCS = "docs/reST/ref"

RE_C_FUNCTIONS_AND_CONVENTIONS = re.compile(
    r"{\"(.*?)\".*?(METH_.*?),.*?}", flags=re.DOTALL
)
RE_STUBS = re.compile(
    r"(    @overload)?\n(?:    )?def (.*?)(\(.*?\)) ->", flags=re.DOTALL
)
RE_SIGNATURES = re.compile(r"sg:`(.*?)(\(.*?\)).*?`")

for mod_name, c_file, stub_file, doc_file in modules:
    try:
        assert os.path.exists(os.path.join(SRC_C, c_file))
        assert os.path.exists(os.path.join(STUBS, stub_file))
        assert os.path.exists(os.path.join(DOCS, doc_file))
    except AssertionError:
        print(mod_name)
        raise

for mod_name, c_file, stub_file, doc_file in modules:
    if use_specific is not None and use_specific != mod_name:
        continue

    print(mod_name, "testing")

    with open(os.path.join(SRC_C, c_file)) as file:
        c_data = file.read()
    functions_and_conventions = RE_C_FUNCTIONS_AND_CONVENTIONS.findall(
        c_data
    )  # [("func_name", "METH_..."), ...]
    functions = []  # ["func_name", ...]
    for name, convention in functions_and_conventions:
        if (
            "KEYWORD" not in convention
            and "NOARGS" not in convention
            and not name.startswith("_")
        ):
            functions.append(name)

    with open(os.path.join(STUBS, stub_file)) as file:
        stub_data = file.read()
    stubs = RE_STUBS.findall(
        stub_data
    )  # [("overload_or_emtpy", "func_name", "(signature_with_newlines)"), ...]

    with open(os.path.join(DOCS, doc_file)) as file:
        doc_data = file.read()
    doc_signatures = RE_SIGNATURES.findall(
        doc_data
    )  # [("func_name", "(signature)"), ...]

    stub_funcs = functions.copy()
    last_name = None
    last_overload = ""
    for overload, name, signature in stubs:
        if name not in stub_funcs or name in ignore_stub_names:
            continue
        signature = signature[:-1].strip().replace("\n", "")
        try:
            idx = -1 if signature[-1] != "," else -2
            assert signature[idx] == "/"
        except AssertionError:
            print(f"{stub_file}:", name)
            raise
        if overload != "":
            last_overload = overload
            if last_name != name:
                if last_name is not None:
                    stub_funcs.remove(last_name)
                last_name = name
            continue
        if last_overload != "":
            stub_funcs.remove(last_name)
            last_overload = ""
            last_name = None
        stub_funcs.remove(name)

    doc_funcs = functions.copy()
    last_name = None
    for name, signature in doc_signatures:
        if name not in doc_funcs or name in ignore_doc_names:
            continue
        signature = signature[:-1]
        try:
            if (
                name not in ignore_uppercase_params
                and mod_name not in ignore_uppercase_params_modules
            ):
                assert all(
                    word.split("=")[0].islower()
                    for word in signature.split(",")
                    if "/" not in word
                )
            assert signature[-1] == "/"
        except AssertionError:
            print(f"{doc_file}:", name)
            raise

        if last_name != name:
            if last_name is not None:
                doc_funcs.remove(last_name)
            last_name = name

    print(mod_name, "passed")

[MyreMylar]

Just to surface it for you:

running stubcheck
error: pygame.camera.AbstractCamera.get_image is inconsistent, stub argument "self" should be positional or keyword (remove leading double underscore)
Stub: in file /home/runner/work/pygame-ce/pygame-ce/buildconfig/stubs/pygame/camera.pyi:25
def (pygame.camera.AbstractCamera, Union[pygame.surface.Surface, None] =) -> pygame.surface.Surface
Runtime: in file /home/runner/.local/lib/python3.10/site-packages/pygame/camera.py:36
def (self, dest_surf=None)

error: pygame.camera.AbstractCamera.get_image is inconsistent, stub argument "dest_surf" should be positional or keyword (remove leading double underscore)
Stub: in file /home/runner/work/pygame-ce/pygame-ce/buildconfig/stubs/pygame/camera.pyi:25
def (pygame.camera.AbstractCamera, Union[pygame.surface.Surface, None] =) -> pygame.surface.Surface
Runtime: in file /home/runner/.local/lib/python3.10/site-packages/pygame/camera.py:36
def (self, dest_surf=None)

Found 2 errors (checked [21](https://github.com/pygame-community/pygame-ce/actions/runs/6242696075/job/16947022011?pr=2457#step:6:22)5 modules)
Stubcheck failed.

[Starbuck5]

@Matiiss what's the status of this PR?

Now that Python 3.7 has been dropped this should be able to move forward, once you fix the merge conflict.

[Matiiss]

I haven't done much about this for a long while now, to paint a picture of what is missing besides fixing that error, it's pretty much all of these functions that should be "fixed" and lowercasing all the parameter names in the docs, because that is how the docs used to indicate them being positional-only.

[Starbuck5]

I haven't done much about this for a long while now, to paint a picture of what is missing besides fixing that error, it's pretty much all of these functions that should be "fixed" and lowercasing all the parameter names in the docs, because that is how the docs used to indicate them being positional-only.

I had to read this sentence several times to understand it 😄

To do (I think):

• Fix merge conflict
• Fix stub errors
• "Fix" this list of random functions that are in an ignore list for some reason
• Lowercase the parameter names in the docs (they're already in lowercase??)

[MyreMylar]

LGTM.

Don't think I've ever looked at so many forward slashes on one diff.

[Starbuck5]

Skimming over this it looks good, a good straightening out of our patterns to match modern conventions.

I am interested to make sure @ankith26 is okay with this approach before we merge.

[ankith26]

LGTM, thanks a lot for this PR 🎉