Cannot set A.__new__ back to object.__new__ once it is set to something else.

[Paebbels]

You cannot set A.__new__ back to object.__new__ once it is set to something else.

class A:
    def __init__(self, arg):
        pass

A.__new__ = object.__new__


A(123) # it passes

A.__new__ = 123 # you can pass anything here
A.__new__ = object.__new__

A(123) # it cannot pass
# TypeError: object.__new__() takes exactly one argument (the type to instantiate)

The cause of this is analyzed here: #105888 (comment)

Edited for clarity.

Original Post

Summary

If the __new__ method of a class was replaced by a dummy method and later replaced by it original __new__, the method has a divergent behavior as before. Before and and after the swap, both methods point to the same physical address, so according to my debugging capabilities from within Python code, both methods are identical, but show a different behavior.

As I understood the answer to my StackOverflow question, CPython implements a "forgiving" feature, if only one of the __new__ or __init__ methods was modified by a user e.g. expecting additional arguments. So the other method ignores them, because __call__ of the meta-class passes args and kwargs to __new__ as well as __init__.

So far so good, but why does __new__ behave differently in my case if the methods are still identical according the function pointer locations? I my opinion, some internal checks in CPython are not correct and the exception is raised inadvertently.

Full reproducer file

Background

I'm implementing several features like abstract methods, abstract classes, must-override methods, singletone behavior, slotted classes (automatic inference of slots) and mixin classes (deferred slots) using a user-defined meta-class called ExtendedType. The following code can be found as a whole at pyTooling/pyTooling on the development branch.

Depending on the internal algorithms of ExtendedType, it might decide a class A is abstract. If so, the __new__ method is replaced by a dummy method raising an exception (AbstractClassError). Later, when a class B(A) inherits from A, the meta-class might come to the decision, B isn't abstract anymore, thus we want to allow the object creation again and allow calling for the original __new__ method. Therefore, the original method is preserved as a field in the class (__orig_new__).

Replacing __new__ was choosen, because it has no performance impact compared to overriding __call__ in the meta-class, as this is called on every object creation, also on non-abstract classes.
(Performance loss is 1:2.47 on CPython 3.10.11 and 1:2.86 on CPython 3.11.3 measured by pytest-benchmark)

To simplify the internal algorithms for the abstractness decision, the meta-class implements a boolean named-parameter abstract.

Bug Report

class AbstractClassError(Exception):
    pass

class M(type):
  # staticmethod
  def __new__(cls,  className, baseClasses, members, abstract):
    newClass = type.__new__(cls, className, baseClasses, members)
        
    if abstract:
      def newnew(cls, *_, **__):
        raise AbstractClassError(f"Class is abstract")
            
      # keep original __new__ and exchange it with a dummy method throwing an error
      newClass.__new_orig__ = newClass.__new__
      newClass.__new__ = newnew
    else:
      # 1. replacing __new__ with original (preserved) method doesn't work
      newClass.__new__ = newClass.__new_orig__

    return newClass


class A(metaclass=M, abstract=True):
  pass


class B(A, abstract=False):
  def __init__(self, arg):
    self.arg = arg

b = B(5)

This causes an error message:

TypeError: object.__new__() takes exactly one argument (the type to instantiate)

Investigate if swapping methods worked correctly:

print("object.__new__ ", object.__new__)
print("A.__new_orig__ ", A.__new_orig__)
print("A.__new__      ", A.__new__)
print("B.__new__      ", B.__new__)

Results:

object.__new__  <built-in method __new__ of type object at 0x00007FFE30EDD0C0>
A.__new_orig__  <built-in method __new__ of type object at 0x00007FFE30EDD0C0>
A.__new__       <function M.__new__.<locals>.newnew at 0x000001CF11AE5A80>
B.__new__       <built-in method __new__ of type object at 0x00007FFE30EDD0C0>

So, the preserved method in __new_orig__ is identical to object.__new__ and is again the same after swapping back the __new__ method in class B.

Behavior of Ordinary Classes

Here is a comparison with two ordinary classes X and Y(X):

class X:
    pass

class Y(X):
    def __init__(self, arg):
        self.arg = arg

y = Y(3)

Of cause this will work, but are the new methods different?

object.__new__  <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
A.__new_orig__  <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
A.__new__       <function M.__new__.<locals>.newnew at 0x000001CD1FB459E0>
B.__new__       <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
X.__new__       <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
Y.__new__       <built-in method __new__ of type object at 0x00007FFE3B61D0C0>

Also X and Y use the same __new__ method as B or object.

Comparing results:

print("Y.__new__      ", Y.__new__)
y = Y(3)
print("y.arg          ", y.arg)

print("B.__new__      ", B.__new__)
b = B(5)
print("b.arg          ", y.arg)
Results:

Outputs:

Y.__new__       <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
y.arg           3
B.__new__       <built-in method __new__ of type object at 0x00007FFE3B61D0C0>
Traceback (most recent call last):
  File "C:\Temp\newIstKomisch.py", line 67, in <module>
    b = B(5)
        ^^^^
TypeError: object.__new__() takes exactly one argument (the type to instantiate)

Further Reading:

• Full reproducer file
• Source code in CPython
• Question on Stackoverflow

Your environment

• CPython versions tested on: 3.11.3
• Operating system and architecture: Windows 11 64-bit, x86-64

---

/cc @skoehler

[Paebbels]

Implementing that __call__ method in the meta-class fixes the bug in CPython with a huge performance impact.

But it shows according to the view from with the Python code, both __new__ methods are identical.

def __call__(cls, *args, **kwargs):
  if cls.__new__ is object.__new__:
    newCls = cls.__new__(cls)
  else:
    newCls = cls.__new__(cls, *args, **kwargs)

  cls.__init__(newCls, *args, **kwargs)

  return newCls

[sunmy2019]

Update: Ignore this post since it is partially incorrect.

---

The behavior of object.__new__ is depending on C level argument type->tp_new.

Once it is set during class creation, there is no easy way from Python side to change its value. The subsequent stores of __new__ will only affect B.__dict__, not tp_new.

cpython/Objects/typeobject.c

Lines 5412 to 5446 in 4426279

	static int
	object_init(PyObject *self, PyObject *args, PyObject *kwds)
	{
	PyTypeObject *type = Py_TYPE(self);
	if (excess_args(args, kwds)) {
	if (type->tp_init != object_init) {
	PyErr_SetString(PyExc_TypeError,
	"object.__init__() takes exactly one argument (the instance to initialize)");
	return -1;
	}
	if (type->tp_new == object_new) {
	PyErr_Format(PyExc_TypeError,
	"%.200s.__init__() takes exactly one argument (the instance to initialize)",
	type->tp_name);
	return -1;
	}
	}
	return 0;
	}
	static PyObject *
	object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
	{
	if (excess_args(args, kwds)) {
	if (type->tp_new != object_new) {
	PyErr_SetString(PyExc_TypeError,
	"object.__new__() takes exactly one argument (the type to instantiate)");
	return NULL;
	}
	if (type->tp_init == object_init) {
	PyErr_Format(PyExc_TypeError, "%.200s() takes no arguments",
	type->tp_name);
	return NULL;
	}
	}

I agree it is a bug.

1. I don't think it would be an easy change since this behavior has been long-standing for at least 10 years.

2. And I don't think it is worth a change since it seems like a corner case.

3. But I also don't object to a change if there is enough reason for it, and the performance is not dropping (too much)

[sunmy2019]

To circumvent it, avoid setting __new__ or __init__ after class creation.

[Paebbels]

To circumvent it, avoid setting new or init after class creation.

I'll try and check what I can do. Maybe I can handover the modified or swapped methods to type.__new__(...).

I'll also investigate another workaround by adding a wrapper for __new__ if the method is swapped back, which removes unwanted parameters from a call to the original object.__new__. Maybe the performance impact is not so drastic.

[Paebbels]

For your example:

cls.__new__ = cls.__new__

Where can I find the C code for the assignment, which adds an entry to cls.__dict__ and replaces tp_new with a redirection?

When __new__ is a (function) slot, why is it converted to a __dict__ entry? If a user defines __slots__ then accesses are not converted to __dict__ entries, right? Or is there a difference for attribute slots vs. function slots?

Another problem:
How should this behavior be portable to e.g. PyPy? CPython is a reference implementation but also a language specification. So PyPy would need to implement the same bug.

An assignment like the one above shouldn't break the code.
It's not a question of corner cases and niches, but about mathematical correctness of a programming language, right?!

---

I also would like to understand (still need to search it) how @abstract in ABCMeta works. It might have a similar problem if it works like my approach.

[Paebbels]

Just a curious side question:
I wrote my meta-class ExtendedType adding several features to Python (3.7..11), some are enhancing the speed in Python 3.10 by 25% or still 5-10% in Python 3.11.

Core features:

• Slots are inferred automatically from type annotations, no need to manually set __slots__.
  • Detection of wrongly used attributes, if not declared with type annotations.
  • 4x lower memory footprint
  • faster code execution
• Correct behavior of slots in multiple inheritance scenarios with deferred slots (marking classes as mixin-classes)
• Implementation of abstract methods, no need for ABCMeta.
  (To my knowledge: ABCMeta is incompatible with user defined meta-classes in multiple inheritance scenarios. Moreover features provided by multiple meta-classes like abstract and singleton, can't be composed if not provided by a single meta-class)
  • Improved exception messages.
• Implementation of mustoverride methods - similar to abstract methods.
• Implementation of singletons.

I have unit tests (pytest) to check the behavior and performance of my code.

I would be open in donating that to CPython so type could be enhanced with low performance impact. Especially if implemented by CPython in C instead of Python code.

I'm not sure whom to contact by issues or email and speak to.

[Paebbels]

@sunmy2019 I fixed your example 😃:

The original exampled posted in the morning missed an __init__ method in A.

Assigning something different then object.__new__:

class M(type):
  # staticmethod
  def __new__(cls,  className, baseClasses, members):
    newClass = type.__new__(cls, className, baseClasses, members)
    oldnew = newClass.__new__
    newClass.__new__ = 5
    newClass.__new__ = oldnew
    return newClass

class A(metaclass=M):
  def __init__(self, arg):
    pass

a = A(5)

Error message:

Traceback (most recent call last):
  File "C:\Temp\newIsDoof.py", line 15, in <module>
    a = A(5)
        ^^^^
TypeError: object.__new__() takes exactly one argument (the type to instantiate)

It can be fixed like that:

class M(type):
  # staticmethod
  def __new__(cls,  className, baseClasses, members):
    newClass = type.__new__(cls, className, baseClasses, members)
    oldnew = newClass.__new__
#    newClass.__new__ = 5
    newClass.__new__ = oldnew
    return newClass

So the assignment operator has a hidden magic feature if the assign method to tp_new is object.__new__ then it accepts the new method, otherwise it does wired stuff.

[sunmy2019]

For your example:

My example is wrong so I deleted it.

[sunmy2019]

Update: Ignore this post since it is partially incorrect.

---

Where can I find the C code for the assignment, which adds an entry to cls.__dict__ and replaces tp_new with a redirection?

I want to clarify this.

slot tp_new is assigned during class creation. So if the class is created with a customized __new__, then it can never use object.__new__ again. (otherwise, you will meet the bug above)

You can find the assignment here.

cpython/Objects/typeobject.c

Lines 6245 to 6284 in 3bb0994

	static int
	type_ready_set_new(PyTypeObject *type)
	{
	PyTypeObject *base = type->tp_base;
	/* The condition below could use some explanation.
	It appears that tp_new is not inherited for static types whose base
	class is 'object'; this seems to be a precaution so that old extension
	types don't suddenly become callable (object.__new__ wouldn't insure the
	invariants that the extension type's own factory function ensures).
	Heap types, of course, are under our control, so they do inherit tp_new;
	static extension types that specify some other built-in type as the
	default also inherit object.__new__. */
	if (type->tp_new == NULL
	&& base == &PyBaseObject_Type
	&& !(type->tp_flags & Py_TPFLAGS_HEAPTYPE))
	{
	type->tp_flags |= Py_TPFLAGS_DISALLOW_INSTANTIATION;
	}
	if (!(type->tp_flags & Py_TPFLAGS_DISALLOW_INSTANTIATION)) {
	if (type->tp_new != NULL) {
	// If "__new__" key does not exists in the type dictionary,
	// set it to tp_new_wrapper().
	if (add_tp_new_wrapper(type) < 0) {
	return -1;
	}
	}
	else {
	// tp_new is NULL: inherit tp_new from base
	type->tp_new = base->tp_new;
	}
	}
	else {
	// Py_TPFLAGS_DISALLOW_INSTANTIATION sets tp_new to NULL
	type->tp_new = NULL;
	}
	return 0;
	}

---

adds an entry to cls.__dict__ and replaces tp_new with a redirection

It adds an entry? Yes.
replaces tp_new with a redirection? No. No need for that. __dict__ has higher precedence over tp_new.

---

You can also verify that '__new__' is not in B.__dict__ during creation, and tp_new is the same with A's.

After you do B.__new__ = ..., you can verify '__new__' is in B.__dict__

class M(type):
    # staticmethod
    def __new__(cls, className, baseClasses, members, abstract):
        newClass = type.__new__(cls, className, baseClasses, members)

        if abstract:

            def newnew(*_, **__):
                ...

            # keep original __new__ and exchange it with a dummy method throwing an error
            newClass.__new_orig__ = newClass.__new__
            newClass.__new__ = newnew
        else:
            # 1. replacing __new__ with original (preserved) method doesn't work
            print("before __new__", newClass, newClass.__dict__)
            newClass.__new__ = newClass.__new_orig__
            print("after  __new__", newClass, newClass.__dict__)

        return newClass


class A(metaclass=M, abstract=True):
    ...


class B(A, abstract=False):
    ...


B()

before __new__ <class '__main__.B'> {'__module__': '__main__', '__doc__': None}
after  __new__ <class '__main__.B'> {'__module__': '__main__', '__doc__': None, '__new__': <built-in method __new__ of type object at 0x560a4ca48400>}

[Paebbels]

I got your point about newClass.__new__ = ... creating the dictionary item the first time. Now attribute lookup order kicks in. But why does

class M(type):
  # staticmethod
  def __new__(cls,  className, baseClasses, members):
    newClass = type.__new__(cls, className, baseClasses, members)
    oldnew = newClass.__new__
    newClass.__new__ = oldnew
    return newClass

not fail?
It's an assignment and __new__ is not yet in the __dict__. Following your reasoning, it was then not added to __dict__ otherwise it would fail. Then would in turn mean, the assignment was not executed, right?

---

Also following you explanations, tp_new is not changed, but an entry is added to __dict__. If tp_new wasn't touched, why does this check fail? According to your reasoning, an earlier item in the lookup order was added, so tp_new isn't reached anymore. That can't be true.

[skoehler]

I want to clarify this.

slot tp_new is assigned during class creation. So if the class is created with a customized __new__, then it can never use object.__new__ again. (otherwise, you will meet the bug above)

I don't follow. Let's consider the following code again:

class M(type):
  # staticmethod
  def __new__(cls,  className, baseClasses, members):
    newClass = type.__new__(cls, className, baseClasses, members)
    oldnew = newClass.__new__
    newClass.__new__ = 5
    newClass.__new__ = oldnew
    return newClass

class A(metaclass=M):
  def __init__(self, arg):
    pass

a = A(7)

If what you say was true, then the assignment of 5 wouldn't matter. The class has already been created at that point by type.__new__(). So how does the assignment of 5 change the outcome?\

Also, when commenting out the assignment of 5, how come the assignment of oldnew doesn't trigger the same effect?

Alternatively, if the creation of the class happens later, i.e., after the completion of M.__new__(), how does the assignment of 5 affect the outcome since it is later overwritten by oldnew?

We established that example fails if and only if the assignment of 5 is not commented out. Nothing you said so far seems explain that. May I ask you to clarify further?

The evidence we presented strongly suggest that assignments to __new__ have some side-effect that happens at the time of assignment!

[sunmy2019]

If tp_new wasn't touched,

It is touched on here.

cpython/Objects/typeobject.c

Lines 4420 to 4435 in 3bb0994

	static PyObject *
	object_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
	{
	if (excess_args(args, kwds)) {
	if (type->tp_new != object_new) {
	PyErr_SetString(PyExc_TypeError,
	"object.__new__() takes exactly one argument (the type to instantiate)");
	return NULL;
	}
	if (type->tp_init == object_init) {
	PyErr_Format(PyExc_TypeError, "%.200s() takes no arguments",
	type->tp_name);
	return NULL;
	}
	}

[sunmy2019]

@Paebbels @skoehler You both are correct. I am missing something. I dig a bit deeper.

---

For this code:

class M(type):
  # staticmethod
  def __new__(cls,  className, baseClasses, members):
    newClass = type.__new__(cls, className, baseClasses, members)
    oldnew = newClass.__new__
    newClass.__new__ = oldnew
    return newClass

class A(metaclass=M):
  def __init__(self, arg):
    pass

a = A(5)

The following things happened.

1. type.__new__ create a class, and set its C slot tp_new to object_new.

cpython/Objects/typeobject.c

Lines 6245 to 6284 in 3bb0994

	static int
	type_ready_set_new(PyTypeObject *type)
	{
	PyTypeObject *base = type->tp_base;
	/* The condition below could use some explanation.
	It appears that tp_new is not inherited for static types whose base
	class is 'object'; this seems to be a precaution so that old extension
	types don't suddenly become callable (object.__new__ wouldn't insure the
	invariants that the extension type's own factory function ensures).
	Heap types, of course, are under our control, so they do inherit tp_new;
	static extension types that specify some other built-in type as the
	default also inherit object.__new__. */
	if (type->tp_new == NULL
	&& base == &PyBaseObject_Type
	&& !(type->tp_flags & Py_TPFLAGS_HEAPTYPE))
	{
	type->tp_flags |= Py_TPFLAGS_DISALLOW_INSTANTIATION;
	}
	if (!(type->tp_flags & Py_TPFLAGS_DISALLOW_INSTANTIATION)) {
	if (type->tp_new != NULL) {
	// If "__new__" key does not exists in the type dictionary,
	// set it to tp_new_wrapper().
	if (add_tp_new_wrapper(type) < 0) {
	return -1;
	}
	}
	else {
	// tp_new is NULL: inherit tp_new from base
	type->tp_new = base->tp_new;
	}
	}
	else {
	// Py_TPFLAGS_DISALLOW_INSTANTIATION sets tp_new to NULL
	type->tp_new = NULL;
	}
	return 0;
	}

2. the assignment of newClass.__new__ = oldnew does affect the C slot. (I was wrong about it)

You can see it here. The update_slot is supposed to update tp_new to the correct value (a slot_tp_new to indicate looking through the class's __dict__)

cpython/Objects/typeobject.c

Lines 4927 to 4938 in 4d140e5

	if (res == 0) {
	/* Clear the VALID_VERSION flag of 'type' and all its
	subclasses. This could possibly be unified with the
	update_subclasses() recursion in update_slot(), but carefully:
	they each have their own conditions on which to stop
	recursing into subclasses. */
	PyType_Modified(type);
	if (is_dunder_name(name)) {
	res = update_slot(type, name);
	}
	assert(_PyType_CheckConsistency(type));

But it does not. It special cases object.__new__ here. In this case, its tp_new is left unchanged.

cpython/Objects/typeobject.c

Lines 9814 to 9835 in 4d140e5

	else if (Py_IS_TYPE(descr, &PyCFunction_Type) &&
	PyCFunction_GET_FUNCTION(descr) ==
	_PyCFunction_CAST(tp_new_wrapper) &&
	ptr == (void**)&type->tp_new)
	{
	/* The __new__ wrapper is not a wrapper descriptor,
	so must be special-cased differently.
	If we don't do this, creating an instance will
	always use slot_tp_new which will look up
	__new__ in the MRO which will call tp_new_wrapper
	which will look through the base classes looking
	for a static base and call its tp_new (usually
	PyType_GenericNew), after performing various
	sanity checks and constructing a new argument
	list. Cut all that nonsense short -- this speeds
	up instance creation tremendously. */
	specific = (void *)type->tp_new;
	/* XXX I'm not 100% sure that there isn't a hole
	in this reasoning that requires additional
	sanity checks. I'll buy the first person to
	point out a bug in this reasoning a beer. */
	}

3. When this class is created, it used tp_new. For a class with tp_new = object_new, it calls the C function object_new. For a class with tp_new = slot_tp_new, it goes through class's __dict__.

So in this example.

class M(type):
  # staticmethod
  def __new__(cls,  className, baseClasses, members):
    newClass = type.__new__(cls, className, baseClasses, members)
    # now tp_new = object_new 
    oldnew = newClass.__new__
    newClass.__new__ = oldnew
    # due to the special case in `update_slot`,
    # `tp_new` is unchanged
    # everything is fine.
    return newClass

class M(type):
  # staticmethod
  def __new__(cls,  className, baseClasses, members):
    newClass = type.__new__(cls, className, baseClasses, members)
    # now tp_new = object_new 
    oldnew = newClass.__new__

    newClass.__new__ = 5
    # tp_new = slot_tp_new (indicating looking through `newClass.__dict__`)

    newClass.__new__ = oldnew
    # due to the special case in `update_slot`,
    # `tp_new` is left unchanged (`= slot_tp_new`)
    # and `newClass.__dict__` is updated. Still looking through `__dict__`.
 
    return newClass

A()
# looking through `A.__dict__`
# found `object.__new__`
# object_new checks `tp_new` of A
# Finally fails here.

[sunmy2019]

Super weird bug. Looks like we should fix the update_slot logic.

[Paebbels]

I couldn't follow all the steps yet. I might need to read the C code a few more times to puzzle all parts together. But we know for sure, something is odd in that behavior.

Btw. what's the sentence about the beer? Am I and @skoehler qualified for it?

[sunmy2019]

something is odd in that behavior.

Unfortunately, the 23-yr old slot updating code was introduced in Python 2.3 721f62e. There are codes relying on this behavior. So at least this part is not changeable.

The argument check code is added 12 yrs ago. 96384b9

@Paebbels Will you allow me to edit your post? I want to state the reason and the problem more clearly.

Btw. what's the sentence about the beer? Am I and @skoehler qualified for it?

You should ask @gvanrossum. But I guess it was correct when that line was written. Somewhere else broke the things.

[Paebbels]

@sunmy2019, yes you can modify the post.

[sunmy2019]

I personally think it is the current design does not allow setting A.__new__ in object.__new__ in any case.

class A:
    def __new__(cls, arg):
        return super().__new__(cls)
    
    def __init__(self, arg):
        pass


A(123) # it passes

A.__new__ = object.__new__

A(123) # it cannot pass
# TypeError: object.__new__() takes exactly one argument (the type to instantiate)

I prefer to make it clear in the doc: do not assign A.__new__ = object.__new__.

[Paebbels]

Ok, now we have understood it.
But is it the intension of Python not allowing it to be set?

What group of developers is deciding on such a question?

I don't think so.

[sunmy2019]

What group of developers is deciding on such a question?

I think core developers can decide if there are no controversies. Otherwise we require a PEP.

is it the intension of Python

It's just my guess. It may never be never supported. https://godbolt.org/z/eE81aGPMo