Optimize namedtuple creation

[methane]

BPO	28638
Nosy	@gvanrossum, @rhettinger, @ncoghlan, @pitrou, @vstinner, @ericvsmith, @giampaolo, @methane, @serhiy-storchaka, @MojoVampire, @llllllllll, @zhangyangyu, @JelleZijlstra, @lazka, @ethanhs
PRs	bpo-28638: speed up namedtuple creation by avoiding exec #2736 bpo-28638: Optimize namedtuple() creation time by minimizing use of exec() #3454
Files	28638-functools-no-namedtuple.patch namedtuple-no-compile.patch namedtuple1.py namedtuple-clinic.diff namedtuple-clinic2.diff functools-CacheInfo-Makefile.patch namedtuple-clinic3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2017-09-10.17:25:13.282>
created_at = <Date 2016-11-08.04:07:24.207>
labels = ['3.7', 'library', 'performance']
title = 'Optimize namedtuple creation'
updated_at = <Date 2017-09-10.17:25:13.282>
user = 'https://github.com/methane'

bugs.python.org fields:

activity = <Date 2017-09-10.17:25:13.282>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2017-09-10.17:25:13.282>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2016-11-08.04:07:24.207>
creator = 'methane'
dependencies = []
files = ['45386', '45388', '45399', '45578', '45580', '45724', '45738']
hgrepos = []
issue_num = 28638
keywords = ['patch']
message_count = 62.0
messages = ['280277', '280279', '280282', '280283', '280284', '280285', '280288', '280291', '280297', '280298', '280300', '280303', '280356', '280543', '280560', '280561', '281336', '281339', '281356', '282172', '282178', '282182', '282278', '282279', '282412', '285615', '298400', '298444', '298453', '298457', '298482', '298485', '298486', '298487', '298488', '298489', '298490', '298491', '298493', '298499', '298500', '298503', '298514', '298515', '298566', '298570', '298571', '298574', '298581', '298601', '298630', '298631', '298637', '298641', '298648', '298653', '298670', '298681', '298730', '301700', '301804', '301819']
nosy_count = 15.0
nosy_names = ['gvanrossum', 'rhettinger', 'ncoghlan', 'pitrou', 'vstinner', 'eric.smith', 'giampaolo.rodola', 'methane', 'serhiy.storchaka', 'josh.r', 'llllllllll', 'xiang.zhang', 'JelleZijlstra', 'lazka', 'ethan smith']
pr_nums = ['2736', '3454']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue28638'
versions = ['Python 3.7']

[methane]

I surprised how functools make import time slower.
And I find namedtuple makes it slower.

When I replaced

_CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"])

this line with _CachedInfo._source:

(before)
$ ~/local/py37/bin/python3 -m perf timeit -s 'import importlib, functools' -- 'importlib.reload(functools)'
.....................
Median +- std dev: 1.21 ms +- 0.01 ms

(replaced)
$ ~/local/py37/bin/python3 -m perf timeit -s 'import importlib, functools' -- 'importlib.reload(functools)'
.....................
Median +- std dev: 615 us +- 12 us

[methane]

I feel this patch is safe enough to be landed in 3.6.

[zhangyangyu]

I doubt this deserves a change. The slow import is the case only the first time functools is imported. Later imports will just use the cache (sys.modules). And if this is gonna change, maybe we don't have to copy the entire namedtuple structure?

[methane]

The slow import is the case only the first time functools is imported. Later imports will just use the cache (sys.modules).

Yes. But first import time is also important for CLI applications.
That's why mercurial and Bazaar has lazy import system.

Since many stdlib uses functools, many applications may be suffered from
slow functools import even if we remove it from site.py.

maybe we don't have to copy the entire namedtuple structure?

https://docs.python.org/3.5/library/functools.html#functools.lru_cache

The doc says it's a namedtuple. So it should be namedtuple compatible.

[zhangyangyu]

Yes. But first import time is also important for CLI applications.
That's why mercurial and Bazaar has lazy import system.

The lazy import system could benefit many libs so the result could be impressive. But here only functools is enhanced, half a millisecond is reduced.

Performance of course is important, but replicating code sounds not good. It means you have to maintain two pieces.

[methane]

The lazy import system could benefit many libs so the result could be impressive. But here only functools is enhanced, half a millisecond is reduced.

On the other hand, implementing lazy import makes application complex.

This patch only enhance functools, but it is very important one module.
Even if we remove functools from site.py, most applications relies on it,
especially for functools.wraps().
This patch can optimize startup time of them.

Half milliseconds is small, but it isn't negligible on some situation.
Some people loves tools quickly starts. For example, there are many
people maintain their vimrc to keep <50~100ms startup time.
And Python is common language to implement vim plugins.

Additionally, it make noise when profiling startup time.
I've very confused when I saw PyParse_AddToken() in profile.
Less noise make it easy to optimize startup time.

Performance of course is important, but replicating code sounds not good. It means you have to maintain two pieces.

Yes. Balance is important.
I want to hear more opinions from more other devs.

[serhiy-storchaka]

What is the main culprit, importing the collections module or compiling a named tuple?

[serhiy-storchaka]

Using namedtuple is not new in 3.6, thus this is not a regression that can be fixed at beta stage.

Inlining the source of a named tuple class looks ugly solution. It would be better to write the source in separate file and import it. Makefile can have a rule for recreating this source file if collections.py is changed.

More general solution would be to make namedtuple() using cached precompiled class and update the cache if it doesn't match namedtuple arguments.

Yet one solution is to make namedtuple() not using compiling, but return patched local class. But Raymond is against this.

[methane]

What is the main culprit, importing the collections module or compiling a named tuple?

In this time, later.
But collections module takes 1+ ms to import too.
I'll try to optimize it.

Using namedtuple is not new in 3.6, thus this is not a regression that can be fixed at beta stage.

Make sense.

More general solution would be to make namedtuple() using cached precompiled class and update the cache if it doesn't match namedtuple arguments.

What "precompiled class" means? pyc file? or source string to be
executed?

Yet one solution is to make namedtuple() not using compiling, but return patched local class. But Raymond is against this.

I'll search the discussion.
I think anther solution is reimplement namedtuple by C.
As far as I remember, attrs [1] does it.

[1] https://pypi.python.org/pypi/attrs

[serhiy-storchaka]

Here is a sample patch that make namedtuple() not using dynamic compilation. It has rough the same performance effect as inlining the named tuple source, but affects all named tuples.