Unloading extension modules not always safe

[etrepum]

BPO	838140
Nosy	@jackjansen, @etrepum, @ronaldoussoren

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/jackjansen'
closed_at = <Date 2004-07-15.22:29:12.000>
created_at = <Date 2003-11-07.20:44:30.000>
labels = ['OS-mac']
title = 'Unloading extension modules not always safe'
updated_at = <Date 2004-07-15.22:29:12.000>
user = 'https://github.com/etrepum'

bugs.python.org fields:

activity = <Date 2004-07-15.22:29:12.000>
actor = 'jackjansen'
assignee = 'jackjansen'
closed = True
closed_date = None
closer = None
components = ['macOS']
creation = <Date 2003-11-07.20:44:30.000>
creator = 'bob.ippolito'
dependencies = []
files = []
hgrepos = []
issue_num = 838140
keywords = []
message_count = 9.0
messages = ['18936', '18937', '18938', '18939', '18940', '18941', '18942', '18943', '18944']
nosy_count = 3.0
nosy_names = ['jackjansen', 'bob.ippolito', 'ronaldoussoren']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue838140'
versions = ['Python 2.3']

[etrepum]

I will look into the solution for this, but the "for now"
solution would be to never try and unlink bundles because
they may contain ObjC code. For several reasons, ObjC
classes can never be unloaded, so bundles containing ObjC
code can also never be unloaded. This is more than a
problem for just PyObjC, because any arbitrary module may
contain some ObjC code. We need to detect this before
running NSUnlinkModule. I'll try and put together a patch
sometime soon if nobody else does, but for now see: http://
sourceforge.net/tracker/?
func=detail&aid=832025&group_id=14534&atid=114534

[jackjansen]

Logged In: YES
user_id=45365

Bob, I'm confused. As far as I know Python never unloads
extension modules...

[etrepum]

Logged In: YES
user_id=139309

it does if you del sys.modules['somemodule'] and somemodule's
reference count goes to zero.

[jackjansen]

Logged In: YES
user_id=45365

I'm surprised that it does the unload:-)

I think the correct solution would be for the module itself (or
someone close to it) to stash away a reference. As this is only a
problem for some modules (those containing ObjC code) I don't
think a general change is in order.

The real problem is that the "last reference" as Python sees it isn't
really the last reference: the ObjC runtime also has references to
stuff in there.

[jackjansen]

Logged In: YES
user_id=45365

I think that either the module itself, or the package responsible for
the module, should forestall unloading (by tucking away an extra
reference somewhere).

The root of the problem is that the Python refcount doesn't reflect
the all references to the module: ObjC keeps references too.

OTOH: if all modules containing ObjC code have this problem, and
it is easy to detect whether a module contains ObjC code then
adding a safety net to the import code might be a prudent course
of action.

[ronaldoussoren]

Logged In: YES
user_id=580910

Note that the problem doesn't really have anything to do with
unloading modules, the problem occurs when trying to unload a
dylib that doesn't contain the proper init function. See bug
bpo-848907 (sorry about that one, I didn't look for earlier bug
reports).

What happens:

• pydoc calls imp.load_module('__temp__', open('_objc.so'), ...)
• load_module loads _objc.so
• load_module tries to find init__temp__ in _objc.so
• nothing is found and therefore load_module calls NSUnlinkModule
• NSUnlinkModule aborts the process because _objc.so contains
  Objective-C definitions

No code in the extension module has any change to fix things
because the init function is never called.

BTW. The problem also exists in a vanilla Python 2.3 installation on
Panther, I cleared /Library/Python2.3 and ran 'pydoc -k hello' and
this also crashed.

[ronaldoussoren]

Logged In: YES
user_id=580910

A quick workaround would be to let pydoc check the type of
module it tries to load, it currently checks if the file is a binary
module by checking "if info and 'b' in info[2]" (pydoc.py:171). If it
would check if the filetype is not C_EXTENSION (3) the problem
would not arise with pydoc (which probably accounts for >99% of
the problematic load_module calls).

[ronaldoussoren]

Logged In: YES
user_id=580910

Another, probably more correct, workaround is to stop alling
NSUnlinkModule.

This function is only called when an invalid dynamic object is
loaded, which should not occur during normal operation. The only
negative effect of not unloading seems to be that the process
image is larger than really necessary.

[jackjansen]

Logged In: YES
user_id=45365

Fixed in dynload_next.c 2.15.