property_descr_get reuse argument tuple

[llllllllll]

BPO	23910
Nosy	@warsaw, @rhettinger, @vstinner, @ericvsmith, @ericsnowcurrently, @serhiy-storchaka, @llllllllll
Files	namedtuple.patch: nameduple in C. namedtuple-indexer.patch namedtuple-indexer-update.patch namedtuple-indexer-with-news.patch property.patch with-static-tuple.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2015-04-30.15:20:53.514>
created_at = <Date 2015-04-10.20:10:37.345>
labels = ['interpreter-core', 'type-feature']
title = 'property_descr_get reuse argument tuple'
updated_at = <Date 2018-10-01.10:11:02.588>
user = 'https://github.com/llllllllll'

bugs.python.org fields:

activity = <Date 2018-10-01.10:11:02.588>
actor = 'vstinner'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2015-04-30.15:20:53.514>
closer = 'rhettinger'
components = ['Interpreter Core']
creation = <Date 2015-04-10.20:10:37.345>
creator = 'llllllllll'
dependencies = []
files = ['38892', '38896', '38905', '38994', '39210', '39216']
hgrepos = []
issue_num = 23910
keywords = ['patch']
message_count = 26.0
messages = ['240447', '240448', '240449', '240450', '240451', '240452', '240465', '240535', '240565', '240568', '240934', '242082', '242085', '242089', '242091', '242093', '242103', '242112', '242114', '242123', '242124', '242234', '242235', '242273', '243981', '326783']
nosy_count = 8.0
nosy_names = ['barry', 'rhettinger', 'vstinner', 'eric.smith', 'python-dev', 'eric.snow', 'serhiy.storchaka', 'llllllllll']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23910'
versions = ['Python 3.5']

[llllllllll]

I am working on implementing nameduple in C; I am almost there; however, on the path of moving to full compatibility, I ran into a refcount issue somewhere. Hopefully someone can help me work this out.

To describe the issue, When I run the collections tests I most frequently segfault in a non namedtuple test; however, sometimes it runs (and passes) however that probably means I am using an unowned obect somewhere. I am new to the CPython API so I would love to go through this with someone to learn more about how it all works.

I am currently at PyCon and would absolutly love to speak to someone in person. I will be at CPython sprints for at least one day.

[ericvsmith]

What's the motivating use case for this?

[llllllllll]

Ideally, namedtuple is used to make your code cleaner, using "magic" indecies is less clear than using a named index in a lot of cases. Because namedtuple is mainly to make the code more readable, I don't think that it should have an impact on the runtime performance of the code. This means that namedtuple should be a very thin wrapper around tuple. Currently, namedtuple named item access is much slower than elementwise access. I have this as a standalone package (there are some changes in the diff I posted to acheive full backwards compat) here https://pypi.python.org/pypi/cnamedtuple/0.1.5 that show some profiling runs of this code. The notable one to me is named item access being around twice as fast.

Another issue we ran into at work is that we found ways to get the exec call in nametuple to execute arbitrary code; while this would not be a concern for most people by the nature of the way we got this to happen, we wanted to look at other ways of writing namedtuple. I looked through some older discussion and found that non-exec solutions were rejected in the past for perfomance reasons.

[ericvsmith]

Have you thought of just exposing Object/structseq.c?

Before you put much time into this, I'd get Raymond's acceptance of whatever approach you want to take. It might be best to raise it on python-ideas.

[llllllllll]

would the idea be to deprecate namedtuple in favor of a public structseq that is exposed through collections, or change structseq to fit the namedtuple API?

[ericvsmith]

I haven't seen thought it through, just that it seems very similar to a C namedtuple.

[llllllllll]

I stripped down the patch to only the descriptor like we had discussed.

[rhettinger]

Can you post before and afters timings of the patch?

[llllllllll]

# Original version / new python implementation
./python -m timeit -s "from collections import namedtuple;a = namedtuple('a', 'a b c')(1, 2, 3)" "a.a"
10000000 loops, best of 3: 0.07 usec per loop

# C implementation
./python -m timeit -s "from collections import namedtuple;a = namedtuple('a', 'a b c')(1, 2, 3)" "a.a"
10000000 loops, best of 3: 0.028 usec per loop

The fallback is the same implementation that is currently used so this should have no affect on pypi.

[llllllllll]

sorry, I meant pypy

[llllllllll]

I am updating the patch to include an entry in Misc/NEWS.

[rhettinger]

FWIW, the current property(itemgetter(index)) code has a Python creation step, but the actual attribute lookup and dispatch is done entirely in C (no pure python steps around the eval lookup).

Rather than making a user visible C hack directly to namedtuple, any optimization effort should be directly at improving the speed of property() and itemgetter().

Here are some quick notes to get you started:

* The overhead of property() is almost nothing.
* The code for property_descr_get() in Objects/descrobject.c
* It has two nearly zero cost predictable branches
* So the the only real work is a call to 
  PyObject_CallFunctionObjArgs(gs->prop_get, obj, NULL);
* which then calls both
       objargs_mktuple(vargs) 
  and 
       PyObject_Call(callable, args, NULL);
* That can be sped-up by using 
       PyTuple_New(1)
  and a direct call to  PyObject_Call()
* The logic in PyObject_Call can potentially be tightened
  in the context of a property(itemgetter(index)) call.
  Look to see whether recursion check is necessary
  (itemgetter on a tuple subclass that doesn't extend __getitem__
   is non-recursive)
* If so, then entire call to PyObject_Call() in property 
  can potentially be simplified to:
   result = (*call)(func, arg, kw);

I haven't looked too closely at this, but I think you get the general idea. If the speed of property(itemgetter(index)) is the bottleneck in your code, the starting point is to unwind the exact series of C steps performed to see if any of them can be simplified. For the most part, the code in property() and itemgetter() were both implemented using the simplest C parts of the C API rather than the fastest. The chain of calls isn't specialized for the common use case (i.e. property_get() needing exactly 1 argument rather than a variable length arg tuple and itemgetter doing a known integer offset on a list or tuple rather than the less common case of generic types and a possible tuple of indices).

We should start by optimizing what we've already got. That will have a benefit beyond named tuples (faster itemgetters for sorting and faster property gets for the entire language).

It also helps us avoid making the NT code less familiar (using custom private APIs rather than generic, well-known components).

It also reduces the risk of breaking code that relies on the published implementation of named tuple attribute lookups (for example, I've seen deployed code that customizes the attribute docstrings like this):
Point = namedtuple('Point', ['x', 'y'])
Point.x = property(Point.x.fget, doc='abscissa')
Point.y = property(Point.y.fget, doc='ordinate')
coordinate = Point(x=250, y=43)