Add CodeType.replace() method

[vstinner]

BPO	37032
Nosy	@vstinner, @blueyed, @Carreau, @pablogsal
PRs	bpo-37032: Add CodeType.replace() method #13542 bpo-37032: Document CodeType.replace (new in python 3.8) #17776 [3.8] bpo-37032: Document CodeType.replace (GH-17776) #17779

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-27.07:45:56.864>
created_at = <Date 2019-05-24.09:25:55.826>
labels = ['interpreter-core', '3.8']
title = 'Add CodeType.replace() method'
updated_at = <Date 2020-01-01.06:12:20.408>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-01-01.06:12:20.408>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-27.07:45:56.864>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2019-05-24.09:25:55.826>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 37032
keywords = ['patch']
message_count = 15.0
messages = ['343360', '343370', '343371', '343373', '343374', '343377', '343378', '343381', '343384', '343434', '343435', '343589', '343610', '344816', '344817']
nosy_count = 4.0
nosy_names = ['vstinner', 'blueyed', 'mbussonn', 'pablogsal']
pr_nums = ['13542', '17776', '17779']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37032'
versions = ['Python 3.8']

[vstinner]

Each type types.CodeType constructor changes, a lot of applications break. Python 3.8 added a new parameter which broke for example the Genshi project, just to name one.

I propose to add a new CodeType.replace() method, similar to datetime.datetime.replace and namedtuple._replace() for example:

$ python3
Python 3.7.3 (default, Mar 27 2019, 13:41:07) 

>>> import datetime
>>> d=datetime.datetime.now()
>>> d2=d.replace(year=2010)
>>> d, d2
(datetime.datetime(2019, 5, 24, 11, 25, 3, 839966), datetime.datetime(2010, 5, 24, 11, 25, 3, 839966))

>>> import collections
>>> Point = collections.namedtuple("Point", "x y")
>>> p=Point(1, 2)
>>> p2=p._replace(y=3)
>>> p, p2
(Point(x=1, y=2), Point(x=1, y=3))

[vstinner]

Python 3.8 added a new parameter which broke for example the Genshi project

Pablo fixed it in Genshi:
edgewall/genshi#19

[vstinner]

See also bpo-37034: "Argument Clinic omits name of keyword-only parameter on _PyArg_BadArgument() call".

[vstinner]

Fix in Hypothesis:
HypothesisWorks/hypothesis@8f47297#diff-a097452b8ffe8323641ccccac335fbf9

[vstinner]

ipython fix:
ipython/ipython@248128d

[vstinner]

It's not like Python 3.8 is the first type in Python history that code constructor changed.

The PyCode_New() function slowly growed from 3 parameters in 1990 to 16 paramters in 2019 :-)

History of PyCode_New().

New "posonlyargcount" parameter:

commit 8c77b8c
Author: Pablo Galindo <Pablogsal@gmail.com>
Date: Mon Apr 29 13:36:57 2019 +0100

bpo-36540: PEP-570 -- Implementation (GH-12701)

New "kwonlyargcount" parameter, bpo-1549670:

commit 4f72a78
Author: Guido van Rossum <guido@python.org>
Date: Fri Oct 27 23:31:49 2006 +0000

Jiwon Seo's PEP-3102 implementation.
See SF#1549670.
The compiler package has not yet been updated.

New "freevars" and "cellvars" parameters, PEP-227:

commit 64949cb
Author: Jeremy Hylton <jeremy@alum.mit.edu>
Date: Thu Jan 25 20:06:59 2001 +0000

PEP-227 implementation
...
[Include/compile.h](https://github.com/python/cpython/blob/main/Include/compile.h)
    Add co_freevars and co_cellvars slots to code objects.
    Update PyCode_New() to take freevars and cellvars as arguments

New "firstlineno" and "lnotab" parameters:

commit da4eb5c
Author: Guido van Rossum <guido@python.org>
Date: Fri Jan 24 03:43:35 1997 +0000

Instead of emitting SET_LINENO instructions, generate a line number
table which is incorporated in the code object.  This way, the runtime
overhead to keep track of line numbers is only incurred when an
exception has to be reported.

New "stacksize" parameter:

commit 8b993a9
Author: Guido van Rossum <guido@python.org>
Date: Fri Jan 17 21:04:03 1997 +0000

Add co_stacksize field to codeobject structure, and stacksize argument
to PyCode_New() argument list.  Move MAXBLOCKS constant to conpile.h.

Added accurate calculation of the actual stack size needed by the
generated code.

Also commented out all fprintf statements (except for a new one to
diagnose stack underflow, and one in #ifdef'ed out code), and added
some new TO DO suggestions (now that the stacksize is taken of the TO
DO list).

New "argcount", "nlocals", "flags" and "varnames" parameters:

commit 681d79a
Author: Guido van Rossum <guido@python.org>
Date: Tue Jul 18 14:51:37 1995 +0000

keyword arguments and faster calls

New "name" parameter:

commit 9bfef44
Author: Guido van Rossum <guido@python.org>
Date: Mon Mar 29 10:43:31 1993 +0000

...
* Added function name to code object.  Print it for code and function
  objects.  THIS MAKES THE .PYC FILE FORMAT INCOMPATIBLE (the version
  number has changed accordingly)
...

New "filename" parameter:

commit 3f5da24
Author: Guido van Rossum <guido@python.org>
Date: Thu Dec 20 15:06:42 1990 +0000

"Compiling" version

The very first version of the function was:

static codeobject *
newcodeobject(code, consts, names)
object *code;
object *consts;
object *names;

From this commit:

commit 10dc2e8
Author: Guido van Rossum <guido@python.org>
Date: Sun Nov 18 17:27:39 1990 +0000

Initial revision

[vstinner]

It's not like Python 3.8 is the first type in Python history that code constructor changed.

Oops, typo: It's not like Python 3.8 is the first *time* in Python history that the code constructor changed.

[vstinner]

Fix in Cloud Pickle:
cloudpipe/cloudpickle@b9dc17f

[vstinner]

See bpo-36886 "Failed to construct CodeType on Python-3.8.0a4" for other failures caused by the addition of the "posonlyargcount" parameter to code constructor.

[vstinner]

New changeset a9f05d6 by Victor Stinner in branch 'master':
bpo-37032: Add CodeType.replace() method (GH-13542)
a9f05d6

[vstinner]

Once Python 3.8 beta1 will be released, it would be interesting to patch all projects that I had to be fixed to handle the new "posonlyargcount" to use the new "replace()" method is available.

Maybe even remove the fix for Python 3.8 alpha 4 and only use use replace() on Python 3.8 and newer.

[pablogsal]

Is there anything more on this issue?

[vstinner]

Pablo: "Is there anything more on this issue?"

I proposed "Once Python 3.8 beta1 will be released, it would be interesting to patch all projects that I had to be fixed to handle the new "posonlyargcount" to use the new "replace()" method is available."

But that only concerns third party project,s no Python directly, so I close the issue :-)

[blueyed]

Thanks for this great improvement!

Was about to suggest having something more futureproof when noticing this with pdbpp, found this, and submitted PRs to use it for ipython and hypothesis also.

[vstinner]

submitted PRs to use it for ipython and hypothesis also

Cool! Thank you for doing that ;-)

[feisuzhu]

Found this great improvement years after my frustration, many thanks for this work!

[vstinner]

Found this great improvement years after my frustration, many thanks for this work!

I'm glad that it helped you ;-)