Modify the _struct module to use FASTCALL and Argument Clinic

[vstinner]

BPO	29300
Nosy	@vstinner, @larryhastings, @methane, @skrah, @vadmium, @serhiy-storchaka
Files	struct_fastcall.patch struct_fastcall-2.patch struct_fastcall-3.patch struct_fastcall-4.patch struct_fastcall-5.patch struct_fastcall-6.patch struct_fastcall-7.patch cache_struct.patch unpack_buffer.patch cache_struct-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-02-10.11:33:37.652>
created_at = <Date 2017-01-17.16:31:45.553>
labels = ['3.7', 'expert-argument-clinic', 'performance']
title = 'Modify the _struct module to use FASTCALL and Argument Clinic'
updated_at = <Date 2017-02-10.11:33:37.651>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-02-10.11:33:37.651>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-02-10.11:33:37.652>
closer = 'vstinner'
components = ['Argument Clinic']
creation = <Date 2017-01-17.16:31:45.553>
creator = 'vstinner'
dependencies = []
files = ['46319', '46422', '46431', '46432', '46434', '46436', '46488', '46489', '46490', '46492']
hgrepos = []
issue_num = 29300
keywords = ['patch']
message_count = 48.0
messages = ['285663', '285669', '286326', '286327', '286329', '286346', '286348', '286349', '286350', '286351', '286357', '286358', '286364', '286367', '286373', '286645', '286659', '286712', '286756', '286761', '286766', '286767', '286768', '286770', '286771', '286772', '286773', '286774', '286775', '286776', '286781', '286782', '286783', '286784', '286785', '286786', '286789', '286790', '286791', '286792', '286795', '286799', '286800', '286834', '286934', '286943', '287125', '287517']
nosy_count = 7.0
nosy_names = ['vstinner', 'larry', 'methane', 'skrah', 'python-dev', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue29300'
versions = ['Python 3.7']

[vstinner]

Attached patch modify the _struct module to use FASTCALL and Argument Clinic.

AC requires a summary line, so I duplicated the first sentence of iter_unpack() docstring:

iter_unpack(fmt, buffer, /)
    Return an iterator yielding tuples unpacked from the given bytes.
    
    Return an iterator yielding tuples unpacked from the given bytes
    source according to the format string, like a repeated invocation of
    unpack_from().  Requires that the bytes length be a multiple of the
    format struct size.

[serhiy-storchaka]

Why not just convert the struct module to Argument Clinic?

[vstinner]

Patch version 2 converts most functions and methods to Argument Clinic, except of unpack() and unpack_into() which require "*args" support in Argument Clinic: see the issue bpo-20291.

[vstinner]

I reformatted docstrings, and wrote a summary line.

[serhiy-storchaka]

It looks to me that the Py_buffer converter can be used for all the buffer parameter and special purposed converter can be used for the fmt parameter.

[vstinner]

Patch version 3:

• Use also Argument Clinic for Struct.__init__()
• Rename fmt to format: in the code and docstring. By the way, Struct docstring was wrong: Struct() accepts a 'format' keyword argument, not 'fmt'
• Use Py_buffer converter for the buffer parameter, except for iter_unpack()
• Document that unpack_from() now accepts keywords and write an unit test
• Fix Struct.__init__() error message: the method accepts also Unicode
• Document that Struct accepts str encodable to ASCII and bytes

I would be nice to use Py_buffer format for the "buffer" argument of iter_unpack(), but Argument Clinic calls PyBuffer_Release(&buffer), whereas the buffer should stay alive after the function exit. Is there a way to clone/copy a Py_buffer?

For the fmt/format argument: s_init() uses custom code to accept Unicode encodable to ASCII and bytes objects. Using Argument Clinic to check fmt type would require to write a converter. I would prefer to not make too many changes in a single patch, and I don't know how to write such converter. I suggest to do that later and keep the existing code. It seems like all functions getting a format ends in s_init() to check the format argument.

[serhiy-storchaka]

The converter for the format parameter already exists. It is cache_struct(). You just need to change its interface.

• Fix Struct.__init__() error message: the method accepts also Unicode
• Document that Struct accepts str encodable to ASCII and bytes

I think this should be backported to older versions. Maybe there is an open issue for this.

I would be nice to use Py_buffer format for the "buffer" argument of iter_unpack(), but Argument Clinic calls PyBuffer_Release(&buffer), whereas the buffer should stay alive after the function exit. Is there a way to clone/copy a Py_buffer?

I agreed that the Py_buffer converter is not applicable here. Just object is good.

[vstinner]

The converter for the format parameter already exists. It is cache_struct(). You just need to change its interface.

The code to convert the format string is in s_init(). The code increases the reference counter, I don't see how to produce the Py_DECREF() code.

Again, I would prefer to do that later. The change is already big enough :-)

I think this should be backported to older versions.

Ok, will do once this change is accepted.

[serhiy-storchaka]

static int
cache_struct_converter(PyObject *arg, PyObject **s_object)
{
    if (arg == NULL) {
        Py_DECREF();
        return 1;
    }
    *s_object = cache_struct(arg); // actually inline this
    if (*s_object == NULL)
        return 0;
    return Py_CLEANUP_SUPPORTED;
}

[serhiy-storchaka]

See also ascii_buffer_converter in Modules/binascii.c for example.

[vstinner]

Thank you for your reviews Serhiy :-) Here is a new patch to take in account your latest comments.

I didn't change the API anymore: Struct.unpack_from() accepts keywords, unpack_from() doesn't.

[vstinner]

See also ascii_buffer_converter in Modules/binascii.c for example.

Nice. I didn't know the "def cleanup" feature. I used it in my patch version 4.