Implement ABCMeta in C

[ilevkivskyi]

BPO	31333
Nosy	@gvanrossum, @warsaw, @brettcannon, @terryjreedy, @scoder, @vstinner, @ned-deily, @methane, @ericsnowcurrently, @serhiy-storchaka, @1st1, @ilevkivskyi, @aaronchall, @miss-islington
PRs	bpo-31333: Re-implement ABCMeta in C #5273 [3.7] bpo-31333: Re-implement ABCMeta in C (GH-5273) #5733 bpo-31333: Fix typo in whatsnew/3.7.rst #5744 [3.7] bpo-31333: Fix typo in whatsnew/3.7.rst (GH-5744) #5745

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-02-18.12:49:52.291>
created_at = <Date 2017-09-03.20:54:21.774>
labels = ['deferred-blocker', '3.8', 'performance', 'extension-modules', '3.7', 'library']
title = 'Implement ABCMeta in C'
updated_at = <Date 2018-02-18.22:35:42.221>
user = 'https://github.com/ilevkivskyi'

bugs.python.org fields:

activity = <Date 2018-02-18.22:35:42.221>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2018-02-18.12:49:52.291>
closer = 'levkivskyi'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2017-09-03.20:54:21.774>
creator = 'levkivskyi'
dependencies = []
files = []
hgrepos = []
issue_num = 31333
keywords = ['patch']
message_count = 18.0
messages = ['301198', '301236', '301241', '301290', '301293', '301301', '301302', '310998', '310999', '312284', '312285', '312287', '312306', '312315', '312323', '312327', '312329', '312331']
nosy_count = 14.0
nosy_names = ['gvanrossum', 'barry', 'brett.cannon', 'terry.reedy', 'scoder', 'vstinner', 'ned.deily', 'methane', 'eric.snow', 'serhiy.storchaka', 'yselivanov', 'levkivskyi', 'Aaron Hall', 'miss-islington']
pr_nums = ['5273', '5733', '5744', '5745']
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue31333'
versions = ['Python 3.7', 'Python 3.8']

[ilevkivskyi]

The idea is that creating ABCs is approximately twice slower than normal classes. Plus there are smaller penalties for various operations with ABCs. This mostly influences the Python interpreter start-up time (because of extensive use of ABCs in importlib), and start-up times of programs that extensively use ABCs.

The situation can be improved by rewriting ABCMeta in C. I have a working implementation, but it is far form being ready and still needs some polishing and optimizations (in particular _abc_cache and friends).

Already at this stage I have one question (I will add more when they appear while I am finishing the implementation): is it OK to make _abc_cache, _abc_negative_cache, _abc_negative_cache_version, and _abc_registry read-only? The point is that I want to prohibit something like this:

    MyABC._abc_cache = "Surprise on updating the cache!"

thus avoiding many PySet_Check(...) calls. These attributes are not documented and start with underscore.

[ericsnowcurrently]

This mostly influences the Python interpreter start-up time
(because of extensive use of ABCs in importlib)

Just to be clear, the only ABCs in importlib are in importlib.abc (and used by importlib.util). This does not impact interpreter startup.

[ilevkivskyi]

Eric,

the only ABCs in importlib are in importlib.abc (and used by importlib.util).
This does not impact interpreter startup.

Hm, indeed, but I see that module 'abc' is in 'sys.modules', probably it is then only used by 'collections.abc'/'_collections_abc'. This means that the influence of 'ABCMeta' on interpreter start-up time will be smaller than I thought. But still start-up times for projects that actively use ABCs will be influenced by 'ABCMeta'.

But what do you think about making private ABC caches read-only attributes? (They are not documented)

[gvanrossum]

I've heard many anecdotal complaints about the lack of speed of ABCs. Even if it doesn't affect core Python startup, it does affect startup of apps, and if people are afraid to use them because of this, it's reasonable to try to do something about it.

[methane]

Hm, indeed, but I see that module 'abc' is in 'sys.modules', probably it is then only used by 'collections.abc'/'_collections_abc'. This means that the influence of 'ABCMeta' on interpreter start-up time will be smaller than I thought. But still start-up times for projects that actively use ABCs will be influenced by 'ABCMeta'.

Since os.environ uses _collections_abc.MutableMapping, importing _collections_abc is not avoidable while startup.

io module uses ABC and it's used for sys.std(io|err|in). It's not avoidable too.

And as you know, typing module will be very common rapidly :)
Even if it doesn't affect "python_startup" microbench, it's important.

[scoder]

Since the number of applications that get along without any file access is probably close to irrelevant, "os" and "io" feel like sufficiently widely used modules to merit being part of a "usual Python startup" benchmark. Maybe we should add one to the benchmark suite? Anyone up for it?

[methane]

"python_startup_nosite" benchmark imports io module,
and "python_startup" benchmark imports os too.
So no need to additional test for them.

You can see it with python -v -S -c 'pass' and python -v -c 'pass'.

[ned-deily]

There has been a lot of work by a number of people on this feature leading up to the 3.7.0b1. Thank you! On PR 5273, Yury says that he believes the feature is ready to merge but would prefer an extension until 3.7.0b2 and that Guido has agreed. I'll defer to Yury's judgement on this and somewhat reluctantly allow an extension: there's a lot going on here. Let's try to get it in as soon as we can, please!

[1st1]

Let's try to get it in as soon as we can, please!

Thank you, Ned! We'll get it merged in the next few days.