Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

class attribute cache failure (regression) #46186

Closed
doublep mannequin opened this issue Jan 20, 2008 · 24 comments
Closed

class attribute cache failure (regression) #46186

doublep mannequin opened this issue Jan 20, 2008 · 24 comments
Assignees
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker type-bug An unexpected behavior, bug, or error

Comments

@doublep
Copy link
Mannequin

doublep mannequin commented Jan 20, 2008

BPO 1878
Nosy @gvanrossum, @arigo, @birkenfeld, @rhettinger, @jcea, @amauryfa, @benjaminp

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = 'https://github.com/gvanrossum'
closed_at = <Date 2008-08-19.20:14:29.093>
created_at = <Date 2008-01-20.17:40:57.326>
labels = ['interpreter-core', 'type-bug', 'release-blocker']
title = 'class attribute cache failure (regression)'
updated_at = <Date 2008-10-13.11:46:22.763>
user = 'https://bugs.python.org/doublep'

bugs.python.org fields:

activity = <Date 2008-10-13.11:46:22.763>
actor = 'jcea'
assignee = 'gvanrossum'
closed = True
closed_date = <Date 2008-08-19.20:14:29.093>
closer = 'gvanrossum'
components = ['Interpreter Core']
creation = <Date 2008-01-20.17:40:57.326>
creator = '_doublep'
dependencies = []
files = []
hgrepos = []
issue_num = 1878
keywords = []
message_count = 24.0
messages = ['61316', '61317', '61318', '61322', '61323', '61324', '61325', '61327', '61328', '61342', '61363', '61364', '61365', '61381', '61415', '61447', '70483', '70490', '70523', '71466', '71468', '71476', '71479', '71484']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'arigo', 'georg.brandl', 'rhettinger', 'jcea', 'amaury.forgeotdarc', '_doublep', 'benjamin.peterson']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1878'
versions = ['Python 2.6']

@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 20, 2008

I have a regression from Python 2.5 to Python SVN (would-be-2.6). I
believe this because of class attribute caching. The problem shown
below happens because AbstractGCProtector is an extension class. So,
Python interpreter doesn't have a chance to notice set_default() method
below changes a class attribute.

>>> from notify.all import *
>>> original_protector = AbstractGCProtector.default
>>> new_protector = FastGCProtector ()
>>> AbstractGCProtector.default is new_protector
False
>>> AbstractGCProtector.default is original_protector
True

Please note that this a regression. This code works as expected in 2.3
and 2.5.

@doublep doublep mannequin added interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Jan 20, 2008
@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 20, 2008

Eh, disregard that, I missed one line with set_default() call. Still,
the unit test fails...

@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 20, 2008

OK, here it is:

>>> from notify.all import *
>>> original_protector = AbstractGCProtector.default
>>> new_protector = FastGCProtector ()
>>> AbstractGCProtector.set_default (new_protector)
>>> AbstractGCProtector.default is new_protector
False
>>> AbstractGCProtector.default is original_protector
True

It seems that this behaviour is somewhat random. Sometimes the
False/True lines are reversed, as expected. I.e. it seems that
attribute cache is sometimes recomputed...

@birkenfeld
Copy link
Member

It would be very interesting to know what set_default() actually does.

IOW, without the source code of the extension module we can't do
anything about this.

@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 20, 2008

set_default() is a static method to set 'default'. Because of this:

>>> AbstractGCProtector.default = 42
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type
'notify.gc.AbstractGCProtector'

About source code: go to http://download.gna.org/py-notify/ download,
unpack and do './run-tests.py' at top level. One test fails with Python
SVN precisely because of this problem.

@birkenfeld
Copy link
Member

I'm sorry, but I can't get this to run. With a clean 0.1.14 tarball, I get

Building extension...
running build_ext
building 'notify.gc' extension
creating build
creating build/temp.linux-i686-2.5
creating build/temp.linux-i686-2.5/notify
i686-pc-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -fPIC
-I/usr/include/python2.5 -c notify/gc.c -o
build/temp.linux-i686-2.5/notify/gc.o
creating build/lib.linux-i686-2.5
creating build/lib.linux-i686-2.5/notify
i686-pc-linux-gnu-gcc -pthread -shared
build/temp.linux-i686-2.5/notify/gc.o -L/usr/lib -lpython2.5 -o
build/lib.linux-i686-2.5/notify/gc.so
[1] 28189 segmentation fault ~/devel/python/python run-tests.py

when running with a trunk python (note the "2.5" in the paths...)

When I build the extension manually and comment out the building command
in run-tests.py, I get

Note that most of the time is spent in gc.collect() calls, not in this
package

..............................................................Fatal
Python error: Objects/classobject.c:2311 object at 0x82dd2bc has
negative ref count -606348326
[1] 28540 abort ~/devel/python/python run-tests.py

@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 20, 2008

Weird. Does it even run with a stable Python (not trunk)?

@birkenfeld
Copy link
Member

Yes, runs fine with 2.5.

@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 20, 2008

Can you run the pasted script (from the third comment) manually then?
The crash might be related to the bug in question.

@birkenfeld
Copy link
Member

I've now built my trunk python without debugging enabled, and can
reproduce your problem.

Armin: the extension module directly modifies an extension type's
tp_dict -- what should it do instead to make the cache happy?

@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 20, 2008

Even if there is an easy workaround for the extension (or even a fix, if
modifying 'tp_dict' is not legal), I don't think it would be acceptable
to make a backward-incompatible change in 2.6. I mean, sure Py-notify
is by no means a widely-used library, but can you guarantee that no
other extension will get broken?

@amauryfa
Copy link
Member

The issue seems similar to the one we had in ctypes when the method
attribute cache was implemented.

Ctypes was corrected in r59943. Maybe similar changes are needed for
this extension.

For example,
PyDict_SetItemString (AbstractGCProtector_Type.tp_dict, "default",
new_protector)
should be modified like this:
PyObject_SetAttr(AbstractGCProtector_Type, "default", new_protector)

@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 20, 2008

It doesn't help:

ERROR: test_default_property (test._gc.AbstractGCProtectorTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/paul/notify/test/_gc.py", line 59, in test_default_property
    AbstractGCProtector.set_default (original_protector)
TypeError: can't set attributes of built-in/extension type
'notify.gc.AbstractGCProtector'

With this code:

if (PyObject_SetAttrString ((PyObject *) &AbstractGCProtector_Type,
"default", new_protector)
== -1)
return NULL;

@arigo
Copy link
Mannequin

arigo mannequin commented Jan 21, 2008

I don't see in general how the patch can be kept compatible with
extension modules that change the tp_dict of arbitrary types. All I can
think of is a way to be safe against extension modules that only change
the tp_dict of their own non-heap types (i.e. types defined in C). The
method cache is disabled for types that don't have the
Py_TPFLAGS_HAVE_VERSION_TAG flag; so if we would leave this flag out of
Py_TPFLAGS_DEFAULT, non-heap types from extension modules would by
default not use the cache.

I'm not sure about the API for this. Would we then need to put
Py_TPFLAGS_HAVE_VERSION_TAG explicitly on all core types? And about how
to change tp_dict in a way that makes the cache happy - do we need to
turn type_modified() into a public API?

All in all, the lack of abstraction of the C API might kill the idea of
this patch for CPython.

@birkenfeld
Copy link
Member

We can of course add something like in bpo-1229239, which allows type
attributes to be set with PyObject_SetAttr, thereby updating the cache.

@doublep
Copy link
Mannequin Author

doublep mannequin commented Jan 21, 2008

I personally think that this bug is a showstopper for the caching patch
in 2.6. Well, the problem can be deemed insignificant, but it is sure a
break of backward compatibility and, worse yet, it results in _very_
obscure fails. Even if type dictionary changes are not all that common,
I'm sure there are extensions out there that do it.

For Py3k things can be different. I'm not sure what would be the best
way, but at least Py3k is not required to be compatible with 2.x in all
aspects.

@benjaminp
Copy link
Contributor

Ping

@rhettinger
Copy link
Contributor

Guido, what say you, live with it, revert it, or apply
Py_TPFLAGS_HAVE_VERSION_TAG to all core types?

@rhettinger rhettinger assigned gvanrossum and unassigned arigo Jul 31, 2008
@arigo
Copy link
Mannequin

arigo mannequin commented Jul 31, 2008

Maybe there is a better solution along the following line: conditionally
define Py_TPFLAGS_DEFAULT so that when compiling the Python core it
includes the Py_TPFLAGS_HAVE_VERSION_TAG, but when compiling extension
modules it does not. This should ensure compatibility with many
existing extension modules. (It would still break if they play tricks
like modifying the tp_dict of types other than their own.)

In addition, to support the method cache in newer C extension modules:

* C types defined by C extension modules can include the
Py_TPFLAGS_HAVE_VERSION_TAG explicitly to enable the cache;
  • new C API functions should be introduced to give an official way to
    access the tp_dict of a type, e.g. PyType_{Get,Set,Del}DictItemString().

@gvanrossum
Copy link
Member

I like Armin's latest proposal: have Py_TPFLAGS_DEFAULT include
Py_TPFLAGS_HAVE_VERSION_TAG when compiling the core only. ISTR there's
a way to do this, but I can't find it right now.

@gvanrossum
Copy link
Member

Please review the patch here: http://codereview.appspot.com/3005

@gvanrossum
Copy link
Member

Submitted as r65874.

I will block this for 3.0; 3.0 extensions that want to mess with tp_dict
must explicitly disable this flag.

@benjaminp
Copy link
Contributor

Do we want a test?

@gvanrossum
Copy link
Member

Sure, go right ahead.

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interpreter-core (Objects, Python, Grammar, and Parser dirs) release-blocker type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

5 participants