Derby #11: Convert 50 sites to Argument Clinic across 9 files

[larryhastings]

BPO	20180
Nosy	@loewis, @rhettinger, @taleinat, @larryhastings, @rbtcollins, @methane, @skrah, @vadmium, @serhiy-storchaka
PRs	bpo-20180: use Argument Clinic in itertools module where appropriate #4117 bpo-20180: itertools.groupby Argument Clinic conversion #4170 bpo-20180: complete AC conversion of Objects/stringlib/transmogrify.h #8039 bpo-20180: convert most of itertoolsmodule.c to use Argument Clinic #9164 bpo-20180: Simplify char_converter in Argument Clinic. #9828 bpo-20180: Use argument clinic for dict.pop() and dict.popitem() #12792
Files	_collectionsmodule.c.patch: AC conversion of deque.rotate and _count_elements in the collections module unicodeobject.c.patch: AC conversion of Objects/unicodeobject.c transmogrify.h.patch unicodeobject.c_2.patch: AC conversion of Objects/unicodeobject.c; supersedes the previous patch for this file _collectionsmodule.c.patch: AC conversion of Modules/_collectionsmodules.c except deque.init; supersedes the previous patch of the same name deque_init_with_nullable_Py_ssize_t.patch: AC conversion of deque.init in Modules/_collectionsmodule.c; depends on Larry's patch from issue20341 _randommodule.c.patch xxlimited_xxmodule_xxsubtype_AC_conversion.patch: Complete AC conversion of Modules/xxlimited.c, Modules/xxmodule.c and Modules/xxsubtype.c. taleinat.ac_conversion.itertools.c.patch unicodeobject.c.v3.patch: AC conversion of Objects/unicodeobject.c; supersedes the previous patch for this file unicodeobject.c.v4.patch: same as previous, with Serhiy's review comments addressed unicodeobject.c.v5.patch unicodeobject.c.v6.patch unicodeobject.c.v7.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-12.18:56:33.809>
created_at = <Date 2014-01-08.00:16:02.924>
labels = ['extension-modules', 'type-feature', '3.8', 'expert-argument-clinic']
title = 'Derby #11: Convert 50 sites to Argument Clinic across 9 files'
updated_at = <Date 2019-04-12.07:11:33.603>
user = 'https://github.com/larryhastings'

bugs.python.org fields:

activity = <Date 2019-04-12.07:11:33.603>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-12.18:56:33.809>
closer = 'rhettinger'
components = ['Extension Modules', 'Argument Clinic']
creation = <Date 2014-01-08.00:16:02.924>
creator = 'larry'
dependencies = []
files = ['33535', '33578', '33579', '33604', '33620', '33621', '33625', '33627', '33749', '39169', '39190', '39191', '46292', '46297']
hgrepos = []
issue_num = 20180
keywords = ['patch']
message_count = 63.0
messages = ['207636', '208431', '208587', '208589', '208728', '208803', '208804', '208821', '208827', '208830', '208846', '208855', '208861', '208865', '208873', '209450', '209451', '224137', '224761', '241488', '241819', '241923', '241925', '241927', '241930', '241931', '241934', '242482', '247955', '247987', '248513', '248663', '285498', '285500', '285514', '285516', '285554', '285558', '285559', '285561', '285562', '285563', '286435', '304904', '304931', '304935', '304936', '304937', '304972', '304976', '304981', '305409', '319533', '319933', '320830', '321161', '321166', '324948', '325072', '325121', '325167', '332505', '340011']
nosy_count = 10.0
nosy_names = ['loewis', 'rhettinger', 'taleinat', 'larry', 'rbcollins', 'methane', 'skrah', 'python-dev', 'martin.panter', 'serhiy.storchaka']
pr_nums = ['4117', '4170', '8039', '9164', '9828', '12792']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20180'
versions = ['Python 3.8']

[larryhastings]

This issue is part of the Great Argument Clinic Conversion Derby,
where we're trying to convert as much of Python 3.4 to use
Argument Clinic as we can before Release Candidate 1 on January 19.

This issue asks you to change the following bundle of files:
Modules/_collectionsmodule.c: 4 sites
Modules/itertoolsmodule.c: 11 sites
Modules/_randommodule.c: 1 sites
Objects/unicodeobject.c: 14 sites
Modules/xxlimited.c: 4 sites
Modules/xxmodule.c: 5 sites
Modules/xxsubtype.c: 5 sites
Objects/stringlib/find.h: 1 sites
Objects/stringlib/transmogrify.h: 5 sites

Talk to me (larry) if you only want to attack part of a bundle.

For instructions on how to convert a function to work with Argument
Clinic, read the "howto":
http://docs.python.org/dev/howto/clinic.html

[taleinat]

I'm attaching my conversion of deque.rotate and _count_elements in Modules/_collectionsmodule.c.

There remain 3 other functions in that module to be converted.

One is deque_init which returns int; Larry said (on IRC) to hold off with that for now.

The other two are dequeiter_new and dequereviter_new, which use the O! format, which I am unsure how to handle.

[taleinat]

Attaching patch for complete conversion of Objects/unicodeobject.c.

Notes:

• maketrans() was already converted
• converting the lstrip, rstrip and strip methods required a small, non-trivial change to the code, since they used a common function for argument parsing
• the documentation for replace() now correctly reflects the actual function signature
• I rephrased some doc-strings where it was required, and in several places moved some of the text into parameter descriptions

[taleinat]

Attached patch for AC conversion of Objects/stringlib/transmogrify.h.

Converting the functions in this file required changes to Objects/bytesobject.c and Objects/bytearrayobject.c. Those changes are in the patch as well. The conversion of those files is part of bpo-20179 (Derby #10). I'm adding a comment on that issue as well with a reference to this.

[taleinat]

Attaching a new patch for complete conversion of Objects/unicodeobject.c.
This is to replace the previous patch for this file.

This new patch also converts all other possible unicode methods,
including those not using PyArg_* functions. This includes various
methods with no arguments or with one PyObject* argument.

Unicode methods still not converted, probably impossible to convert:

1. The various find methods (index, count, startswith, rfind, etc.),
   because they use a special function
   'stringlib_parse_args_finds_unicode' to parse their arguments
2. format and format_map, since they are defined elsewhere
3. _decimal2ascii, since it is just for debugging

[taleinat]

Attached nearly complete conversion of Modules/_collectionsmodule.c.

I've converted all functions using PyArg_* functions, including __new__ methods, but not deque.__init__.

I'm attaching a separate patch for deque.__init__.

[taleinat]

Attached a patch converting deque.__init__ in Modules/_collectionsmodule.c, which completes the conversion of this file.

This patch depends Larry's first patch in bpo-20341 (Argument Clinic: add "nullable ints"). This functionality is used to simplify the existing code for accepting None for the 'maxlen' argument to deque.__init__. If needed I can supply a patch which does not depend on Larry's patch.

[taleinat]

Attached is a complete conversion of Modules/_randommodule.c. Three _random.Random methods were converted: seed, getrandbits and __new__.

There's was an problem converting _random.Random.__new__. I overcame it cleanly and think this patch is good. Details about the problem follow.

bpo-1486663 made various __new__ methods of built-in classes only do a '_PyArg_NoKeywords' check if the type is the class itself, i.e. not for sub-classes. This is to allow sub-classes to accept keyword arguments.

This means that for classes that do this, like _random.Random, converting using AC makes it so subclasses can't accept keyword arguments (unless the subclasses implement __new__ properly).

Still, it is possible to convert _random.Random.__new__ and also override __new__ in random.Random (random != _random, and random.Random subclasses _random.Random). This way subclasses of the latter can accept keyword arguments. Since anyone sub-classing _random.Random instead of random.Random can be expected to know to override __new__ or switch to random.Random, I think this is a good solution for this case.

The attached patch does as I suggest above and passes all tests in test_random, including 'test_random_subclass_with_kwargs' which checks this issue. Just to be sure, I also ran the entire test suite, and all tests pass.

[taleinat]

Attached complete conversion of Modules/xxlimited.c, Modules/xxmodule.c and Modules/xxsubtype.c.

There are no tests for these and the code is not used by anything AFAIK, so I was extra careful and went over the diffs manually once everything was done.

[skrah]

I think we should probably not convert xx*.c. They are for newcomers
who will likely be confused by the additional information overload.

[taleinat]

I thought AC was how all new modules were supposed to be written. If so, the example modules should definitely use AC!

Also note that once the generated code is in separate files, the code in the origin files will be simpler and clearer.

[skrah]

The vast majority of C modules are in third-party packages. I doubt that external
modules will all use AC. xxmodule.c is about explaining the basic structure of
a C extension. AC adds nothing to this and obfuscates (IMO) what is going on.

[taleinat]

Thanks, Stefan. I hadn't realized that the xx files were targeted mainly a third-party developers. In this case, I can see the sense in not converting them.

I only wish someone had said something *before* I went ahead and converted them...

[skrah]

I only wish someone had said something *before* I went ahead and converted them...

Yeah, sorry. It's also just my personal opinion. BTW, xxmodule did not compile
and there was a warning for xxsubtype. I just mention it in case these are AC
issues.

[larryhastings]

When AC moves from internal-only tool to supported tool we'll want to convert the xx stuff. But yeah I reckon it makes sense to not convert them yet. The list of files is really more "here's the list of stuff to check out and convert if necessary". best judgement is always called for.

[taleinat]

Attached is a complete AC conversion of itertools.

This is a large conversion, so I was extra careful. I even went over the entire diff manually to check for any errors. Compilation runs without warnings and the entire test suite passes.

Notes:

• I didn't convert itertools.repeat due to the ongoing discussion regarding it.
• I didn't convert itertools.islice since its signature is like range() but with an additional first required argument, making it impossible to convert properly.
• I didn't convert several __new__ methods because they treat *args similarly to zip(): itertools.chain, itertools.product, itertools.zip_longest
• I used Larry's "nullable ints" patch (see bpo-20341) for the 'r' parameter to itertools.permutations.
• I converted all of the following methods when defined: __new__ (except for classes mentioned above), __sizeof__, __reduce__, __setstate__, __copy__, __length_hint__

[taleinat]

And at that, I've finished the conversion of this entire Derby group! Woohoo!

(except for itertools.repeat which is pending discussion)

[loewis]

For the unicodeobject changes, see Rietveld. In addition, it produces many cast errors from PyUnicodeObject* to PyObject*. Otherwise, it looks fine - please apply (after these fixes).

[larryhastings]

All the Derby patches should only go into trunk at this point.

[larryhastings]

Sadly, for political reasons, it's best that we not convert collections, itertools, or random for now. I'll update this issue if the situation changes. Sorry about that!

[taleinat]

Attaching new patch for unicodeobject.c, which supersedes the previous patches.

I've changed the name of the parameter of str.join() to "iterable" to match the docs. (In the previous patch I had changed it from "seq" to "iterable_of_strings".)

I also fixed two minor comments on the doc strings from Martin.

With this, AFAIK, unicodeobject.c should be good to go.

[taleinat]

Here's a new patch for Objects/unicodeobject.c with all of Serhiy's suggestions integrated.

[serhiy-storchaka]

The patch causes a lot of warnings like

Objects/unicodeobject.c: In function ‘unicode_title_impl’:
Objects/unicodeobject.c:10581:5: warning: passing argument 1 of ‘_PyUnicode_Ready’ from incompatible pointer type [enabled by default]
if (PyUnicode_READY(self) == -1)
^
Objects/unicodeobject.c:1484:1: note: expected ‘struct PyObject *’ but argument is of type ‘struct PyUnicodeObject *’
_PyUnicode_Ready(PyObject *unicode)
^

[taleinat]

Version of patch with PyUnicodeObject * warnings fixed.

[serhiy-storchaka]

unicodeobject.c.v5.patch LGTM.

[taleinat]

Hurrah!

And that seems to be it for this group, since collections, itertools and random are not to be converted at this point, as well as the Modules/xx*.c files, and the stringlib files probably require overly extensive changes for conversion.

[skrah]

Sadly, for political reasons, it's best that we not convert collections, itertools, or random for now.

Well, there are also technical reasons. For example, when reviewing a huge patch at the beginning of this year, the sections that touched AC took me 10 times longer to review than the rest.

[python-dev]

New changeset 27dc9a1c061e by INADA Naoki in branch 'default':
Issue bpo-20180: convert unicode methods to AC.
https://hg.python.org/cpython/rev/27dc9a1c061e

[methane]

Thank you, Martin and Serhiy.

As http://bugs.python.org/issue20180#msg247987 , remained module to converted
is transmogrify.h.

But docstrings of methods in transmogrify.h were moved to bytes_methods.c in bpo-26765.

May I close this issue for now?

[serhiy-storchaka]

Seems you haven't updated generated AC files.

[python-dev]

New changeset 01b06ca45f64 by INADA Naoki in branch 'default':
Issue bpo-20180: forgot to update AC output.
https://hg.python.org/cpython/rev/01b06ca45f64

[methane]

Uhhh! I'm sorry.

[vadmium]

For str.format_map(mapping), yes the parsing happens in Objects/stringlib/unicode_format.h, but I don’t see that as a big problem. Moving this back to “needs patch”, assuming it is okay to convert format_map().

Other than from that, there are just tricky things left like the str() constructor, str.format(*args, **kwargs) (see bpo-20291), and the shared code in Objects/stringlib/find.h and Objects/stringlib/transmogrify.h. I don’t know what to do about those.

[serhiy-storchaka]

Seems Raymond have changed his mind about using Argument Clinic in itertools (msg302908). Tal, do you mind to update your itertools patch and create a pull request?

[taleinat]

Id be happy to update the itertools patch.

I haven't been following AC work for quite a while. I would expect this to entail:

1. applying the patch to the master branch
2. merging any conflicts
3. running the clinic.py script again
4. review the output and compare it to the previous output to make sure things look okay

Would you also prefer this in the form of a GitHub PR?

[serhiy-storchaka]

All correct. The final patch should be in the form of GitHub PR.

Don't specify the self parameter explicitly, just rename the corresponding variable. The rest of the patch LGTM.

[taleinat]

What about islice? Does AC now support complex enough signatures to support it? If not, should I leave the comment as is?

[serhiy-storchaka]

Keep islice() non-converted. For the r parameter of permutations() use "r: object = None".

[taleinat]

As far as I can tell, with the creation of a separate clinic/itertools.c.h, the typedefs etc. need to be moved to the top of the file so that the #include can come after the typedefs and before the rest of the code which references clinic output. Is this indeed the way to go?

[serhiy-storchaka]

Yes, it is. Actually the #include should be before lists of methods and static PyTypeObject initializers. I think it is better to move all method lists and type initializers to the bottom of the file.

[taleinat]

See PR 4117 on GitHub with the itertools AC conversion.

[taleinat]

To enable progressing with this in smaller steps, I created PR 4170 which converts only itertools.groupby and the intertnal _grouper with minimal required changes.

[taleinat]

IMO we should create new issues for AC conversion of the collections and random modules (assuming they haven't been converted yet), and close this issue.

[methane]

IMO we should create new issues for AC conversion of the collections and random modules (assuming they haven't been converted yet), and close this issue.

I agree with you. Let's finish derby.

[taleinat]

See updated PR for Objects/stringlib/transmogrify.h in #52287.

[taleinat]

New changeset c929df3 by Tal Einat in branch 'master':
bpo-20180: complete AC conversion of Objects/stringlib/transmogrify.h (GH-8039)
c929df3

[taleinat]

Now that transmogrify is done, the only open question is about itertools and collections.

1. Should collections be converted?
2. How to proceed with itertools? There are two PRs ready, with the one just for groupby() approved, but not yet reviewed by Raymond Hettinger.

[rhettinger]

New changeset 3286ce4 by Raymond Hettinger (Tal Einat) in branch 'master':
bpo-20180: itertools.groupby Argument Clinic conversion (GH-4170)
3286ce4

[rhettinger]

New changeset c4bccd3 by Raymond Hettinger (Tal Einat) in branch 'master':
bpo-20180: convert most of itertoolsmodule.c to use Argument Clinic (GH-9164)
c4bccd3

[taleinat]

Now just the collections module is left from this group.

Raymond, what would you like done with it WRT AC conversion, if at all?

[rhettinger]

Now just the collections module is left from this group.
Raymond, what would you like done with it WRT AC conversion,
if at all?

I would like to pass on the collections module. Having looked at the patch, I think the code is currently better as is.

That bring this one Derby to a close. Congratulations :-)

[serhiy-storchaka]

New changeset 65ce60a by Serhiy Storchaka in branch 'master':
bpo-20180: Simplify char_converter in Argument Clinic. (GH-9828)
65ce60a

[methane]

New changeset 9e4f2f3 by Inada Naoki in branch 'master':
bpo-20180: Use argument clinic for dict.pop() and dict.popitem() (GH-12792)
9e4f2f3