Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement PEP 590 #81155

Closed
jdemeyer opened this issue May 20, 2019 · 28 comments
Closed

Implement PEP 590 #81155

jdemeyer opened this issue May 20, 2019 · 28 comments
Labels
3.8 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement

Comments

@jdemeyer
Copy link
Contributor

jdemeyer commented May 20, 2019

BPO 36974
Nosy @vstinner, @benjaminp, @encukou, @ambv, @markshannon, @jdemeyer, @pablogsal, @miss-islington
PRs
  • bpo-36974: document PEP 590 #13450
  • bpo-36974: PEP 590 #13185
  • bpo-36974: remove _PyObject_HasFastCall #13460
  • bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async #13464
  • bpo-36974: handle inheritance of vectorcall protocol #13498
  • bpo-36974: rename _FastCallKeywords -> _Vectorcall #13653
  • bpo-36974: Fix GDB integration #13665
  • bpo-36974: add some assertions for PEP 590 #13682
  • bpo-36974: Make tp_call=PyVectorcall_Call work for inherited types #13699
  • bpo-36027 bpo-36974: Fix "incompatible pointer type" compiler warnings #13758
  • bpo-36974: separate vectorcall functions for each calling convention #13781
  • bpo-36974: expand call protocol documentation #13844
  • bpo-36974: inherit tp_vectorcall_offset unconditionally #13858
  • [3.8] bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858) #14342
  • [3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) #14782
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-07-05.14:03:23.218>
    created_at = <Date 2019-05-20.18:01:27.116>
    labels = ['interpreter-core', 'type-feature', '3.8']
    title = 'Implement PEP 590'
    updated_at = <Date 2020-01-16.13:47:47.654>
    user = 'https://github.com/jdemeyer'

    bugs.python.org fields:

    activity = <Date 2020-01-16.13:47:47.654>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-07-05.14:03:23.218>
    closer = 'petr.viktorin'
    components = ['Interpreter Core']
    creation = <Date 2019-05-20.18:01:27.116>
    creator = 'jdemeyer'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36974
    keywords = ['patch']
    message_count = 28.0
    messages = ['343908', '343922', '343929', '343930', '343932', '343934', '343935', '343967', '343968', '343975', '344016', '344077', '344328', '344333', '344339', '344340', '346366', '346492', '347338', '347340', '347347', '348327', '348804', '349046', '349469', '356453', '359507', '360121']
    nosy_count = 8.0
    nosy_names = ['vstinner', 'benjamin.peterson', 'petr.viktorin', 'lukasz.langa', 'Mark.Shannon', 'jdemeyer', 'pablogsal', 'miss-islington']
    pr_nums = ['13450', '13185', '13460', '13464', '13498', '13653', '13665', '13682', '13699', '13758', '13781', '13844', '13858', '14342', '14782']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue36974'
    versions = ['Python 3.8']

    @jdemeyer jdemeyer added 3.8 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement labels May 20, 2019
    @encukou
    Copy link
    Member

    encukou commented May 29, 2019

    New changeset aacc77f by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-36974: implement PEP-590 (GH-13185)
    aacc77f

    @encukou
    Copy link
    Member

    encukou commented May 29, 2019

    New changeset fecb75c by Petr Viktorin in branch 'master':
    bpo-36974: Fix GDB integration (GH-13665)
    fecb75c

    @pablogsal
    Copy link
    Member

    pablogsal commented May 29, 2019

    BUILDBOT FAILURE REPORT
    =======================

    Builder name: AMD64 Ubuntu Shared 3.x
    Builder url: https://buildbot.python.org/all/#/builders/141/
    Build url: https://buildbot.python.org/all/#/builders/141/builds/1866

    Failed tests
    ------------

    • test_pycfunction (test.test_gdb.PyBtTests)

    Test leaking resources
    ----------------------

    Build summary
    -------------

    == Tests result: FAILURE then FAILURE ==

    405 tests OK.

    10 slowest tests:

    • test_multiprocessing_spawn: 6 min 27 sec
    • test_tools: 5 min 34 sec
    • test_concurrent_futures: 5 min 13 sec
    • test_tokenize: 4 min 43 sec
    • test_lib2to3: 3 min 58 sec
    • test_gdb: 3 min 16 sec
    • test_multiprocessing_forkserver: 2 min 25 sec
    • test_asyncio: 2 min 5 sec
    • test_multiprocessing_fork: 1 min 40 sec
    • test_capi: 1 min 36 sec

    1 test failed:
    test_gdb

    17 tests skipped:
    test_devpoll test_idle test_ioctl test_kqueue test_msilib
    test_ossaudiodev test_startfile test_tcl test_tix test_tk
    test_ttk_guionly test_ttk_textonly test_turtle test_winconsoleio
    test_winreg test_winsound test_zipfile64

    1 re-run test:
    test_gdb

    Total duration: 42 min 45 sec

    Tracebacks
    ----------

    Traceback (most recent call last):
      File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_gdb.py", line 890, in test_pycfunction
        self.assertIn('#2 <built-in method gmtime', gdb_output)
    AssertionError: '#2 <built-in method gmtime' not found in 'Breakpoint 1 (time_gmtime) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n\nBreakpoint 1, time_gmtime (self=<module at remote 0x7ffff7f05fb0>, args=(1,)) at ./Modules/timemodule.c:446\n446\t{\n#6 Frame 0x7ffff7f11a50, for file <string>, line 3, in foo ()\n#12 Frame 0x5555557be620, for file <string>, line 5, in bar ()\n#18 Frame 0x5555557be3f0, for file <string>, line 6, in <module> ()\n'
    
    
    Traceback (most recent call last):
      File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_gdb.py", line 890, in test_pycfunction
        self.assertIn('#2 <built-in method gmtime', gdb_output)
    AssertionError: '#2 <built-in method gmtime' not found in 'Breakpoint 1 (time_gmtime) pending.\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".\n\nBreakpoint 1, time_gmtime (self=<module at remote 0x7ffff7f05fb0>, args=(1,)) at ./Modules/timemodule.c:446\n446\t{\n#6 Frame 0x7ffff7f11a50, for file <string>, line 3, in foo ()\n#12 Frame 0x5555557be590, for file <string>, line 5, in bar ()\n#18 Frame 0x5555557be360, for file <string>, line 6, in <module> ()\n'
    
    

    Current builder status
    ----------------------

    The builder is failing currently

    Commits
    -------

    Other builds with similar failures
    ----------------------------------

    Common commits for all builds:

    @pablogsal
    Copy link
    Member

    pablogsal commented May 29, 2019

    @petr is https://bugs.python.org/issue37090 and #13668 also addressing the buildbot failures?

    @encukou
    Copy link
    Member

    encukou commented May 29, 2019

    No, just #13665 is addressing the failures. And it seems that it's successful -- only the slowest of the failed ones (AMD64 Ubuntu Shared 3.x) is still red.

    bpo-37090 extends the test so it can catch more bugs in the future (but there's no rush to get it in...)

    @pablogsal
    Copy link
    Member

    pablogsal commented May 29, 2019

    Great! Thank you very much for the quick fix for the problem.

    For AMD64 Ubuntu Shared 3.x, the last build was successful:

    https://buildbot.python.org/all/#/builders/141/builds/1870/steps/5/logs/stdio

    @encukou
    Copy link
    Member

    encukou commented May 29, 2019

    All stable buildbots are back to green.

    @encukou
    Copy link
    Member

    encukou commented May 30, 2019

    New changeset 735e8af by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-36974: inherit the vectorcall protocol (GH-13498)
    735e8af

    @encukou
    Copy link
    Member

    encukou commented May 30, 2019

    New changeset c145f3b by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-36974: remove _PyObject_HasFastCall (GH-13460)
    c145f3b

    @encukou
    Copy link
    Member

    encukou commented May 30, 2019

    New changeset 37788bc by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-36974: rename _FastCallKeywords -> _Vectorcall (GH-13653)
    37788bc

    @benjaminp
    Copy link
    Contributor

    benjaminp commented May 31, 2019

    New changeset 530f506 by Benjamin Peterson (Jeroen Demeyer) in branch 'master':
    bpo-36974: tp_print -> tp_vectorcall_offset and tp_reserved -> tp_as_async (GH-13464)
    530f506

    @encukou
    Copy link
    Member

    encukou commented May 31, 2019

    I found an issue in PEP-590:

    When inheriting a heap subclass from a vectorcall class that sets .tp_call=PyVectorcall_Call (as recommended), the subclass does not inherit _Py_TPFLAGS_HAVE_VECTORCALL, and thus PyVectorcall_Call does not work for it.
    Possible solutions come to mind:

    • Inherit tp_vectorcall_offset more normally but handle setting __call__ specially
    • Inherit tp_vectorcall_offset (but not _Py_TPFLAGS_HAVE_VECTORCALL) more normally, and make PyVectorcall_Call ignore _Py_TPFLAGS_HAVE_VECTORCALL

    I'll send a PR for the latter.

    @encukou
    Copy link
    Member

    encukou commented Jun 2, 2019

    New changeset fb9423f by Petr Viktorin in branch 'master':
    bpo-36974: Make tp_call=PyVectorcall_Call work for inherited types (GH-13699)
    fb9423f

    @encukou
    Copy link
    Member

    encukou commented Jun 2, 2019

    New changeset e584cbf by Petr Viktorin in branch 'master':
    bpo-36027 bpo-36974: Fix "incompatible pointer type" compiler warnings (GH-13758)
    e584cbf

    @encukou
    Copy link
    Member

    encukou commented Jun 2, 2019

    New changeset 9e3e06e by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-36974: document PEP-590 (GH-13450)
    9e3e06e

    @encukou
    Copy link
    Member

    encukou commented Jun 2, 2019

    New changeset be718c3 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-36974: add some assertions for PEP-590 (GH-13682)
    be718c3

    @encukou
    Copy link
    Member

    encukou commented Jun 24, 2019

    New changeset a8b27e6 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858)
    a8b27e6

    @encukou
    Copy link
    Member

    encukou commented Jun 25, 2019

    New changeset 26fe6c3 by Petr Viktorin (Miss Islington (bot)) in branch '3.8':
    bpo-36974: inherit tp_vectorcall_offset unconditionally (GH-13858) (GH-14342)
    26fe6c3

    @encukou
    Copy link
    Member

    encukou commented Jul 5, 2019

    New changeset 0d722f3 by Petr Viktorin (Jeroen Demeyer) in branch 'master':
    bpo-36974: separate vectorcall functions for each calling convention (GH-13781)
    0d722f3

    @jdemeyer
    Copy link
    Contributor Author

    jdemeyer commented Jul 5, 2019

    Any objections to closing this?

    @encukou
    Copy link
    Member

    encukou commented Jul 5, 2019

    Buildbots are green, closing.

    Thank you for the implementation!

    @encukou encukou closed this as completed Jul 5, 2019
    @ambv
    Copy link
    Contributor

    ambv commented Jul 23, 2019

    New changeset bf8e82f by Łukasz Langa (Jeroen Demeyer) in branch '3.8':
    [3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) (bpo-14782)
    bf8e82f

    @vstinner
    Copy link
    Member

    vstinner commented Jul 31, 2019

    [3.8] bpo-36974: separate vectorcall functions for each calling convention (GH-13781) (bpo-14782)
    bf8e82f

    FYI this change caused a regression in libcomps with Python 3.8 beta3, whereas it works well with Python 3.8 beta2.

    It's not a bug in Python, but it was a bug in libcomps (already fixed upstream). I just fixed libcomps:
    rpm-software-management/libcomps#50

    This project used the following method descriptors (for module functions):

    {"categories_match", (PyCFunction)PyCOMPS_categories_match, METH_KEYWORDS,
    PyCOMPS_validate__doc__},
    {"environments_match", (PyCFunction)PyCOMPS_envs_match, METH_KEYWORDS,
    PyCOMPS_validate__doc__},

    In Python 3.7, importing the module was just fine: descriptor flags are only checked at the first call to the method.

    In Python 3.8, descriptor flags are checked when the module is imported.

    Fedora bug: https://bugzilla.redhat.com/show_bug.cgi?id=1734777

    The fix is to use the right flags: "METH_VARARGS | METH_KEYWORDS" instead of "METH_KEYWORDS".

    Should we add a note like "if you get a 'SystemError: bad call flags' on import, check the descriptor flags of your functions" in What's New in Python 3.8? Maybe with a link to this issue.
    https://docs.python.org/dev/whatsnew/3.8.html#changes-in-the-c-api

    @jdemeyer
    Copy link
    Contributor Author

    jdemeyer commented Aug 5, 2019

    Should we add a note like "if you get a 'SystemError: bad call flags' on import, check the descriptor flags of your functions" in What's New in Python 3.8?

    A better solution would be to change the error message. We could change it to something like SystemError("bad flags 0x2 for PyCOMPS_categories_match")

    But maybe it's not worth it. Are there many projects that define functions/methods but never call them?

    @vstinner
    Copy link
    Member

    vstinner commented Aug 12, 2019

    'SystemError: bad call flags' error message LGTM, I don't think that it should be changed.

    Are there many projects that define functions/methods but never call them?

    I have no idea.

    @miss-islington
    Copy link
    Contributor

    miss-islington commented Nov 12, 2019

    New changeset 9a13a38 by Miss Islington (bot) (Jeroen Demeyer) in branch 'master':
    bpo-36974: expand call protocol documentation (GH-13844)
    9a13a38

    @encukou
    Copy link
    Member

    encukou commented Jan 7, 2020

    bpo-39245 tracks making the API public in Python 3.9.

    @vstinner
    Copy link
    Member

    vstinner commented Jan 16, 2020

    I created bpo-39361: [C API] Document PyTypeObject.tp_print removal in What's New In Python 3.9.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 interpreter-core Interpreter core (Objects, Python, Grammar, and Parser dirs) type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants