reload() is broken for C extension objects

[knepley]

BPO	1144263
Nosy	@mwhudson, @brettcannon, @birkenfeld, @ericsnowcurrently

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2009-04-05.17:41:15.423>
created_at = <Date 2005-02-19.09:20:13.000>
labels = ['interpreter-core', 'type-bug']
title = 'reload() is broken for C extension objects'
updated_at = <Date 2018-05-16.18:12:15.700>
user = 'https://bugs.python.org/knepley'

bugs.python.org fields:

activity = <Date 2018-05-16.18:12:15.700>
actor = 'eric.snow'
assignee = 'none'
closed = True
closed_date = <Date 2009-04-05.17:41:15.423>
closer = 'georg.brandl'
components = ['Interpreter Core']
creation = <Date 2005-02-19.09:20:13.000>
creator = 'knepley'
dependencies = []
files = []
hgrepos = []
issue_num = 1144263
keywords = []
message_count = 6.0
messages = ['60666', '60667', '60668', '60669', '60670', '85537']
nosy_count = 6.0
nosy_names = ['mwh', 'brett.cannon', 'georg.brandl', 'knepley', 'logistix', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1144263'
versions = ['Python 2.6']

[knepley]

1. A C extension module (foo.so) is imported

      import foo

2. The library foo.so is rebuilt with changes

3. We reload the module

      foo = reload(foo)

The reload() method calls imp.load_dynamic() which
eventually gets down to _PyImport_GetDynLoadFunc().
This just calls dlopen(), which returns the old filehandle.

This problem can be fixed by augmenting imp with
unload_dynamic(), which could easily be implemented in
a _PyImport_GetDynUnloadFunc(), which would just
consult its handles[] array, and call dlclose() on the
appropriate handle. This will work if Python was the
only program to dlopen() foo.so.

[mwhudson]

Logged In: YES
user_id=6656

I don't believe this is quite as easy as you make out. It might be
possible to make something that works for platforms that use dlopen
though...

[knepley]

Logged In: YES
user_id=58554

I am only interested in fixing it for the dlopen() case,
which I thnk subsumes every architecture likely to have this
problem appear. I have no problem fixing it myself, but I
need to get all the CVS stuff setup. And I am not sure how
to easily generate the patch to send (I'm used to BK).

[knepley]

Logged In: YES
user_id=58554

I have coded the proposed solution (changed import.c,
importdl.c, and dynload_shlib.c). I will assemble a test
case for it next week.

[logistix]

Logged In: YES
user_id=699438

I don't know if this is a viable general-purpose fix. Keep in
mind that python tracks everything by references, and
reloading doesn't correct existing bindings. If you recompile it
is going to screw up existing function pointers in
PyCFunction objects. In the example below calling a() will
probably end up executing random code or generting a
memory fault.

>> import select
>> a = select.select

... imaginary recompile ...

>>> reload(select)
<module 'select' from 'C:\Python24\DLLs\select.pyd'>
>>> b = select.select
>>> id(a)
18165472
>>> id(b)
18165476
>>> a() #BOOM!

[birkenfeld]

I agree with Grant. There seems to be no way this can be sanely
implemented with today's infrastructure.

[chdominguez]

This may be crazy but +20 years later I am facing the same issue, and as of python 3.9 the issue still exists. Is there still no reliable way to re-import a compiled python module?