implement PEP 552

[benjaminp]

BPO	31650
Nosy	@warsaw, @vstinner, @benjaminp, @serhiy-storchaka
PRs	bpo-31650: PEP 552 (Deterministic pycs) implementation #4575 bpo-31650: Remove _Py_CheckHashBasedPycsMode global config var #8608

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/benjaminp'
closed_at = <Date 2017-12-12.07:17:05.298>
created_at = <Date 2017-09-30.19:59:56.081>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'implement PEP 552'
updated_at = <Date 2018-08-01.16:18:10.739>
user = 'https://github.com/benjaminp'

bugs.python.org fields:

activity = <Date 2018-08-01.16:18:10.739>
actor = 'vstinner'
assignee = 'benjamin.peterson'
closed = True
closed_date = <Date 2017-12-12.07:17:05.298>
closer = 'benjamin.peterson'
components = ['Interpreter Core']
creation = <Date 2017-09-30.19:59:56.081>
creator = 'benjamin.peterson'
dependencies = []
files = []
hgrepos = []
issue_num = 31650
keywords = ['patch']
message_count = 8.0
messages = ['303423', '307884', '307908', '307909', '308047', '308061', '308096', '322869']
nosy_count = 4.0
nosy_names = ['barry', 'vstinner', 'benjamin.peterson', 'serhiy.storchaka']
pr_nums = ['4575', '8608']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue31650'
versions = ['Python 3.7']

[benjaminp]

Now that PEP-552 is accepted, we'll need an actual implementation. Besides the core import changes, there are some perquisites improvements. For one, our SipHash implementation needs to be changed to accept an arbitrary key. Also, the PEP also introduces the first "long" option to the interpreter, so the getopt implementation will have to be enhanced.

[serhiy-storchaka]

While we are here, can other changes be made?

1. Increase the size of the constant part of the signature. Currently it is only 2 bytes (3rd and 4th bytes are b'\r\n') and it is hard to use them in tools like file.

2. Split the magic number on two parts. The first part encodes backward-incompatible changes and can be updated only in new feature releases. The second part encodes backward compatible changes and can be changed in bugfix releases.

3. Maybe even include the minimal Python version? This would simplify the launcher which needs to map a magic number to Python version.

[benjaminp]

On Sat, Dec 9, 2017, at 01:06, Serhiy Storchaka wrote:

Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment:

While we are here, can other changes be made?

All these suggestions seem fine, but they're not in the PEP and the
change is already large enough.

gps also points out we should expand the timestamp field to 64 bits for
2038 proofing.

[benjaminp]

New changeset 42aa93b by Benjamin Peterson in branch 'master':
closes bpo-31650: PEP-552 (Deterministic pycs) implementation (bpo-4575)
42aa93b

[vstinner]

Failure on Python on s390x RHEL 3.x:

http://buildbot.python.org/all/#/builders/21/builds/340

======================================================================
FAIL: test_source_hash (test.test_imp.ImportTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_imp.py", line 335, in test_source_hash
    self.assertEqual(_imp.source_hash(42, b'hi'), b'\xc6\xe7Z\r\x03:}\xab')
AssertionError: b'Yb\x91||\x91bY' != b'\xc6\xe7Z\r\x03:}\xab'

[benjaminp]

Looks fine to me now? http://buildbot.python.org/all/#/builders/21

[vstinner]

The test failed on s390x RHEL 3.x, PPC64 Fedora 3.x, s390x SLES 3.x, s390x Debian 3.x, s390x RHEL 3.x They are now all green, good!

[vstinner]

New changeset 80b762f by Victor Stinner in branch 'master':
bpo-31650: Remove _Py_CheckHashBasedPycsMode global config var (GH-8608)
80b762f