Implement PEP 590

[jdemeyer]

BPO	36974
Nosy	@vstinner, @benjaminp, @encukou, @ambv, @markshannon, @jdemeyer, @pablogsal, @miss-islington
PRs	bpo-36974: document PEP 590 #13450 bpo-36974: PEP 590 #13185 bpo-36974: remove _PyObject_HasFastCall #13460 bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async #13464 bpo-36974: handle inheritance of vectorcall protocol #13498 bpo-36974: rename _FastCallKeywords -> _Vectorcall #13653 bpo-36974: Fix GDB integration #13665 bpo-36974: add some assertions for PEP 590 #13682 bpo-36974: Make tp_call=PyVectorcall_Call work for inherited types #13699 bpo-36027 bpo-36974: Fix "incompatible pointer type" compiler warnings #13758 bpo-36974: separate vectorcall functions for each calling convention #13781 bpo-36974: expand call protocol documentation #13844 bpo-36974: inherit tp_vectorcall_offset unconditionally #13858 [3.8] bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858) #14342 [3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) #14782

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-07-05.14:03:23.218>
created_at = <Date 2019-05-20.18:01:27.116>
labels = ['interpreter-core', 'type-feature', '3.8']
title = 'Implement PEP 590'
updated_at = <Date 2020-01-16.13:47:47.654>
user = 'https://github.com/jdemeyer'

bugs.python.org fields:

activity = <Date 2020-01-16.13:47:47.654>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-07-05.14:03:23.218>
closer = 'petr.viktorin'
components = ['Interpreter Core']
creation = <Date 2019-05-20.18:01:27.116>
creator = 'jdemeyer'
dependencies = []
files = []
hgrepos = []
issue_num = 36974
keywords = ['patch']
message_count = 28.0
messages = ['343908', '343922', '343929', '343930', '343932', '343934', '343935', '343967', '343968', '343975', '344016', '344077', '344328', '344333', '344339', '344340', '346366', '346492', '347338', '347340', '347347', '348327', '348804', '349046', '349469', '356453', '359507', '360121']
nosy_count = 8.0
nosy_names = ['vstinner', 'benjamin.peterson', 'petr.viktorin', 'lukasz.langa', 'Mark.Shannon', 'jdemeyer', 'pablogsal', 'miss-islington']
pr_nums = ['13450', '13185', '13460', '13464', '13498', '13653', '13665', '13682', '13699', '13758', '13781', '13844', '13858', '14342', '14782']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue36974'
versions = ['Python 3.8']

[encukou]

New changeset aacc77f by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: implement PEP-590 (GH-13185)
aacc77f

[encukou]

New changeset fecb75c by Petr Viktorin in branch 'master':
bpo-36974: Fix GDB integration (GH-13665)
fecb75c

[pablogsal]

BUILDBOT FAILURE REPORT
=======================

Builder name: AMD64 Ubuntu Shared 3.x
Builder url: https://buildbot.python.org/all/#/builders/141/
Build url: https://buildbot.python.org/all/#/builders/141/builds/1866

Failed tests
------------

• test_pycfunction (test.test_gdb.PyBtTests)

Test leaking resources
----------------------

Build summary
-------------

== Tests result: FAILURE then FAILURE ==

405 tests OK.

10 slowest tests:

• test_multiprocessing_spawn: 6 min 27 sec
• test_tools: 5 min 34 sec
• test_concurrent_futures: 5 min 13 sec
• test_tokenize: 4 min 43 sec
• test_lib2to3: 3 min 58 sec
• test_gdb: 3 min 16 sec
• test_multiprocessing_forkserver: 2 min 25 sec
• test_asyncio: 2 min 5 sec
• test_multiprocessing_fork: 1 min 40 sec
• test_capi: 1 min 36 sec

1 test failed:
test_gdb

17 tests skipped:
test_devpoll test_idle test_ioctl test_kqueue test_msilib
test_ossaudiodev test_startfile test_tcl test_tix test_tk
test_ttk_guionly test_ttk_textonly test_turtle test_winconsoleio
test_winreg test_winsound test_zipfile64

1 re-run test:
test_gdb

Total duration: 42 min 45 sec

Tracebacks
----------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_gdb.py", line 890, in test_pycfunction
    self.assertIn('#2 <built-in method gmtime', gdb_output)
AssertionError: '#2 <built-in method gmtime' not found in 'Breakpoint 1 (time_gmtime) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n\nBreakpoint 1, time_gmtime (self=<module at remote 0x7ffff7f05fb0>, args=(1,)) at ./Modules/timemodule.c:446\n446\t{\n#6 Frame 0x7ffff7f11a50, for file <string>, line 3, in foo ()\n#12 Frame 0x5555557be620, for file <string>, line 5, in bar ()\n#18 Frame 0x5555557be3f0, for file <string>, line 6, in <module> ()\n'


Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_gdb.py", line 890, in test_pycfunction
    self.assertIn('#2 <built-in method gmtime', gdb_output)
AssertionError: '#2 <built-in method gmtime' not found in 'Breakpoint 1 (time_gmtime) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n\nBreakpoint 1, time_gmtime (self=<module at remote 0x7ffff7f05fb0>, args=(1,)) at ./Modules/timemodule.c:446\n446\t{\n#6 Frame 0x7ffff7f11a50, for file <string>, line 3, in foo ()\n#12 Frame 0x5555557be590, for file <string>, line 5, in bar ()\n#18 Frame 0x5555557be360, for file <string>, line 6, in <module> ()\n'

Current builder status
----------------------

The builder is failing currently

Commits
-------

• 0c2f930

• aacc77f

Other builds with similar failures
----------------------------------

• https://buildbot.python.org/all/#/builders/21/builds/3076
• https://buildbot.python.org/all/#/builders/21/builds/3077
• https://buildbot.python.org/all/#/builders/21/builds/3078
• https://buildbot.python.org/all/#/builders/21/builds/3079
• https://buildbot.python.org/all/#/builders/21/builds/3080
• https://buildbot.python.org/all/#/builders/13/builds/3087
• https://buildbot.python.org/all/#/builders/13/builds/3088
• https://buildbot.python.org/all/#/builders/13/builds/3089
• https://buildbot.python.org/all/#/builders/13/builds/3090
• https://buildbot.python.org/all/#/builders/85/builds/2883
• https://buildbot.python.org/all/#/builders/85/builds/2884
• https://buildbot.python.org/all/#/builders/85/builds/2885
• https://buildbot.python.org/all/#/builders/85/builds/2886
• https://buildbot.python.org/all/#/builders/141/builds/1869
• https://buildbot.python.org/all/#/builders/176/builds/590
• https://buildbot.python.org/all/#/builders/176/builds/591
• https://buildbot.python.org/all/#/builders/176/builds/592
• https://buildbot.python.org/all/#/builders/176/builds/593
• https://buildbot.python.org/all/#/builders/176/builds/594
• https://buildbot.python.org/all/#/builders/16/builds/3055
• https://buildbot.python.org/all/#/builders/16/builds/3056
• https://buildbot.python.org/all/#/builders/16/builds/3057
• https://buildbot.python.org/all/#/builders/16/builds/3058
• https://buildbot.python.org/all/#/builders/16/builds/3059

Common commits for all builds:

• aacc77f

[pablogsal]

@petr is https://bugs.python.org/issue37090 and #13668 also addressing the buildbot failures?

[encukou]

No, just #13665 is addressing the failures. And it seems that it's successful -- only the slowest of the failed ones (AMD64 Ubuntu Shared 3.x) is still red.

bpo-37090 extends the test so it can catch more bugs in the future (but there's no rush to get it in...)

[pablogsal]

Great! Thank you very much for the quick fix for the problem.

For AMD64 Ubuntu Shared 3.x, the last build was successful:

https://buildbot.python.org/all/#/builders/141/builds/1870/steps/5/logs/stdio

[encukou]

All stable buildbots are back to green.

[encukou]

New changeset 735e8af by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: inherit the vectorcall protocol (GH-13498)
735e8af

[encukou]

New changeset c145f3b by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: remove _PyObject_HasFastCall (GH-13460)
c145f3b

[encukou]

New changeset 37788bc by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: rename _FastCallKeywords -> _Vectorcall (GH-13653)
37788bc

[benjaminp]

New changeset 530f506 by Benjamin Peterson (Jeroen Demeyer) in branch 'master':
bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async (GH-13464)
530f506

[encukou]

I found an issue in PEP-590:

When inheriting a heap subclass from a vectorcall class that sets .tp_call=PyVectorcall_Call (as recommended), the subclass does not inherit _Py_TPFLAGS_HAVE_VECTORCALL, and thus PyVectorcall_Call does not work for it.
Possible solutions come to mind:

• Inherit tp_vectorcall_offset more normally but handle setting __call__ specially
• Inherit tp_vectorcall_offset (but not _Py_TPFLAGS_HAVE_VECTORCALL) more normally, and make PyVectorcall_Call ignore _Py_TPFLAGS_HAVE_VECTORCALL

I'll send a PR for the latter.

[encukou]

New changeset fb9423f by Petr Viktorin in branch 'master':
bpo-36974: Make tp_call=PyVectorcall_Call work for inherited types (GH-13699)
fb9423f

[encukou]

New changeset e584cbf by Petr Viktorin in branch 'master':
bpo-36027 bpo-36974: Fix "incompatible pointer type" compiler warnings (GH-13758)
e584cbf

[encukou]

New changeset 9e3e06e by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: document PEP-590 (GH-13450)
9e3e06e

[encukou]

New changeset be718c3 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: add some assertions for PEP-590 (GH-13682)
be718c3

[encukou]

New changeset a8b27e6 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858)
a8b27e6

[encukou]

New changeset 26fe6c3 by Petr Viktorin (Miss Islington (bot)) in branch '3.8':
bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858) (GH-14342)
26fe6c3

[encukou]

New changeset 0d722f3 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
bpo-36974: separate vectorcall functions for each calling convention (GH-13781)
0d722f3

[jdemeyer]

Any objections to closing this?

[encukou]

Buildbots are green, closing.

Thank you for the implementation!

[ambv]

New changeset bf8e82f by Łukasz Langa (Jeroen Demeyer) in branch '3.8':
[3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) (bpo-14782)
bf8e82f

[vstinner]

[3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) (bpo-14782)
bf8e82f

FYI this change caused a regression in libcomps with Python 3.8 beta3, whereas it works well with Python 3.8 beta2.

It's not a bug in Python, but it was a bug in libcomps (already fixed upstream). I just fixed libcomps:
rpm-software-management/libcomps#50

This project used the following method descriptors (for module functions):

{"categories_match", (PyCFunction)PyCOMPS_categories_match, METH_KEYWORDS,
PyCOMPS_validate__doc__},
{"environments_match", (PyCFunction)PyCOMPS_envs_match, METH_KEYWORDS,
PyCOMPS_validate__doc__},

In Python 3.7, importing the module was just fine: descriptor flags are only checked at the first call to the method.

In Python 3.8, descriptor flags are checked when the module is imported.

Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1734777

The fix is to use the right flags: "METH_VARARGS | METH_KEYWORDS" instead of "METH_KEYWORDS".

Should we add a note like "if you get a 'SystemError: bad call flags' on import, check the descriptor flags of your functions" in What's New in Python 3.8? Maybe with a link to this issue.
https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-c-api

[jdemeyer]

Should we add a note like "if you get a 'SystemError: bad call flags' on import, check the descriptor flags of your functions" in What's New in Python 3.8?

A better solution would be to change the error message. We could change it to something like SystemError("bad flags 0x2 for PyCOMPS_categories_match")

But maybe it's not worth it. Are there many projects that define functions/methods but never call them?

[vstinner]

'SystemError: bad call flags' error message LGTM, I don't think that it should be changed.

Are there many projects that define functions/methods but never call them?

I have no idea.

[miss-islington]

New changeset 9a13a38 by Miss Islington (bot) (Jeroen Demeyer) in branch 'master':
bpo-36974: expand call protocol documentation (GH-13844)
9a13a38

[encukou]

bpo-39245 tracks making the API public in Python 3.9.

[vstinner]

I created bpo-39361: [C API] Document PyTypeObject.tp_print removal in What's New In Python 3.9.