Derby #16: Convert 50 sites to Argument Clinic across 9 files

[larryhastings]

BPO	20184
Nosy	@ncoghlan, @larryhastings, @serhiy-storchaka, @1st1, @shihai1991
PRs	gh-64383: Migrate builtins.vars and builtins.dir to Argument Clinic #18512 bpo-20184: Convert termios to Argument Clinic. #22693
Files	issue20184_builtin_conversion.diff: Approach to ensuring all builtins have signatures issue20184_builtin_conversion_v2.diff: All non-type callables in bltinmodule.c converted or classified issue20184_builtin_conversion_v3.diff: Updated to fix test_pydoc and test_gdb issue20184_builtin_conversion_v4.diff: Updated for AC changes issue20184_builtin_conversion_v5.diff: Additional cleanups in the test case issue20184_builtin_conversion_v6.diff: Updated for AC changes in the lead up to 3.4.0 issue20184_builtin_conversion_v7.diff: Updated to apply to trunk (August 2014) dbm_clinic.patch dbm_clinic_2.patch json_clinic.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-01-08.00:21:44.691>
labels = ['extension-modules', 'type-feature', 'expert-argument-clinic']
title = 'Derby #16: Convert 50 sites to Argument Clinic across 9 files'
updated_at = <Date 2020-10-18.14:54:14.974>
user = 'https://github.com/larryhastings'

bugs.python.org fields:

activity = <Date 2020-10-18.14:54:14.974>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Extension Modules', 'Argument Clinic']
creation = <Date 2014-01-08.00:21:44.691>
creator = 'larry'
dependencies = []
files = ['33696', '33701', '33740', '33860', '33861', '34664', '36385', '39032', '39070', '39292']
hgrepos = []
issue_num = 20184
keywords = ['patch']
message_count = 20.0
messages = ['207642', '209159', '209179', '209203', '209409', '209953', '210218', '210375', '215164', '224765', '225385', '225423', '225424', '225425', '241094', '241226', '241347', '242588', '362008', '378872']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'larry', 'nadeem.vawda', 'python-dev', 'serhiy.storchaka', 'yselivanov', 'shihai1991']
pr_nums = ['18512', '22693']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20184'
versions = ['Python 3.5']

[larryhastings]

This issue is part of the Great Argument Clinic Conversion Derby,
where we're trying to convert as much of Python 3.4 to use
Argument Clinic as we can before Release Candidate 1 on January 19.

This issue asks you to change the following bundle of files:
Modules/faulthandler.c: 7 sites
Modules/_pickle.c: 7 sites
Modules/_lzmamodule.c: 7 sites
Python/bltinmodule.c: 6 sites
Modules/termios.c: 6 sites
Modules/syslogmodule.c: 6 sites
Modules/_dbmmodule.c: 4 sites
Modules/_gdbmmodule.c: 2 sites
Modules/_json.c: 5 sites

Talk to me (larry) if you only want to attack part of a bundle.

For instructions on how to convert a function to work with Argument
Clinic, read the "howto":
http://docs.python.org/dev/howto/clinic.html

[ncoghlan]

I'm going to specifically tackle the "bltinmodule" side of things using a test driven approach: adding a new test to test_inspect that checks all the builtin names have signatures (I'll explicitly exclude ones that are known not to be handled, like range and slice).

[ncoghlan]

Attached patch shows the new test case I'm using to ensure that all callable builtins have signatures, and once we get it that way, it stays that way.

Preliminary goal is signatures for all the non-type objects, and once I get to that point, I'll propose it for review.

[ncoghlan]

More comprehensive patch uploaded - all the non-type callables implemented in bltinmodule.c have been converted or classified with a reason for not being converted yet (see the new test in test_inspect.py for details, as well as the AC 3.4 and AC 3.5 comments in the module itself).

I also cleaned up the docstrings for the builtins I actually changed in the patch. There were a few that had never been properly updated for the Py3k transition.

There are still a couple of test failures with this version - the doctest tests get confused by the fact ord and chr now have a doctest in their docstrings, and test_gdb is definitely not in a happy place (that has always been temperamental, though).

I also just realised the Unicode character in the new ord and chr docstrings could pose a compatibility problem at the C compiler source encoding level, so we may have to reconsider that (even though I was rather happy to sneak that obscure Monty Python reference in there).

[ncoghlan]

Assigning to myself to make it clear that bltinmodule is the only part of this still under consideration for 3.4.

The test_pydoc and test_gdb failures pointed to real issues with the previous patch:

• the pydoc errors themselves were incidental, indicating that I had added doctests to chr and ord. However, those new doctests used a Unicode character in a C string, which seems like a recipe for portability trouble. I took those doctests out again, and updated the prose docs instead. I like my obscure multilingual Monty Python reference and would like to keep it now I thought of it :)

• the gdb error suggests that gdb is relying on being able to find builtin_id based on its exact signature, including the parameter names. Rather than trying to figure out the full details of that, I've instead partially reverted its conversion to Argument Clinic by disabling the input block and restoring the old parameter names in the function signature. We can change it back to full conversion for 3.5, after AC has the ability to use different names in the Python signature and in the C implementation function.

This does mean we'll want to update the signature by hand once you merge the patch changing the indicators that argument clinic is looking for.

[ncoghlan]

Larry, if this version looks good to you, I'd like to commit it.

• id() is now back to being a properly generated AC function (since AC can now preserve the old C level signature)

• sorted() is partially converted and has a __text_signature__ compatible docstring. However, full conversion will have to wait for the ability to preserve the kwds dict, since that's the API exposed by the list object.

• with the new never-triggered-by-accident AC syntax, the test case now ensures that all the builtins that are expected to *not* expose signature info at this point, don't. As they're converted for 3.5, that will force them to be added to the list of functions that are checked for compatibility.

I'm thinking it's probably worth flagging this new test as a CPython implementation detail test, though.

[ncoghlan]

Larry, I'd like to include the most-builtin-functions signature support for 3.4, but since the AC Derby is over, it's your call as release manager.

If it looks good to you, go ahead and commit it (I won't have time until later in the week), otherwise just drop the priority and punt it to 3.5.

[ncoghlan]

Interpreting the lack of response as "this can wait until 3.5".

[ncoghlan]

Updated the builtins patch for the 3.4.0 Argument Clinic changes. Also moved the issue back to 3.4, since Larry has indicated that AC conversions are in scope for 3.4.x maintenance releases, so long as no public API behaviour changes.

[larryhastings]

All the Derby patches should only go into trunk at this point.

[ncoghlan]

Tweaked patch to apply cleanly to trunk, but realised it has been a while since I looked at the current state of argument clinic:

• I'm not sure if varargs support has been added yet. If it has, "min", "max", "print", "__build_class__" can be converted. If it hasn't, then the comments referring to 3.4 still need to be updated.

• the comments referring to needing groups support in AC aren't quite right, it's optional group support in inspect that's missing. So the comments relating to "range", "slice", "dir", "getattr", "next", "iter", "vars" should be tweaked accordingly

• tweaking the behaviour of round to make it AC friendly should probably go into its own issue.

Also, Larry, there's a note in the the 3.4 "What's New" about additional signature info landing in Python 3.4 maintenance releases. That note should probably be adjusted at this point, since any new signature data will only be available in 3.5.

[python-dev]

New changeset 6219aa966a5f by Nick Coghlan in branch 'default':
Issue bpo-20184: Add signature introspection for 30 of the builtins
http://hg.python.org/cpython/rev/6219aa966a5f

[ncoghlan]

30 of the builtin functions now support signature introspection thanks to Argument Clinic :)

[ncoghlan]

(Well, they'll support it in 3.5...)

[serhiy-storchaka]

Proposed patch converts _dbm and _gdbm modules to Argument Clinic.

[serhiy-storchaka]

The patch updated to the tip.

[python-dev]

New changeset 49910ff21ba5 by Serhiy Storchaka in branch 'default':
Issue bpo-20184: Converted _dbm and _gdbm modules to Argument Clinic.
https://hg.python.org/cpython/rev/49910ff21ba5

[serhiy-storchaka]

Here is a patch that converts 3 functions in the _json module to Argument Clinic. All other functions doesn't fit with Argument Clinic.

[shihai1991]

Looks like builtins.vars and builtins.dir should be migrated to AC too, so I create PR18512.

[serhiy-storchaka]

New changeset 1bcaa81 by Serhiy Storchaka in branch 'master':
bpo-20184: Convert termios to Argument Clinic. (GH-22693)
1bcaa81

[arhadthedev]

Is this metaissue still relevant after nine years? Currently, conversions into AC are handled in smaller, more specific issues so it could be closed.

[arhadthedev]

BTW, the derby has a few more parts:

• Derby #5: Convert 50 sites to Argument Clinic across 3 files #64373
• Derby #8: Convert 28 sites to Argument Clinic across 2 files #64376
• Derby #9: Convert 52 sites to Argument Clinic across 11 files #64377
• Derby #14: Convert 41 sites to Argument Clinic across 5 files #64382
• Derby #16: Convert 50 sites to Argument Clinic across 9 files #64383
• Derby #18: Convert 31 sites to Argument Clinic across 23 files #64385

[carljm]

It looks to me like all of these derby issues would be better closed. Even if some functions do remain unconverted, it's much more likely at this point that they will be converted because someone notices the unconverted function in the codebase, not because of a derby issue with non-descriptive (and almost certainly now obsolete) name that requires digging through all of the listed modules to see if there's any work remaining to be done on it.

As the derby issues say, it was an effort to convert as much as possible before Python 3.4 release. Python 3.4 is long ago released, so the derby is over.

[carljm]

Will leave this open for a bit to see if a core dev agrees before closing all of them.

[iritkatriel]

Will leave this open for a bit to see if a core dev agrees before closing all of them.

It's been open for long enough now.