sys.modules doc entry should reflect restrictions

[ericsnowcurrently]

BPO	12633
Nosy	@ncoghlan, @merwok, @ericsnowcurrently
Superseder	bpo-28411: [subinterpreters] Eliminate PyInterpreterState.modules.
Files	sys_modules_doc.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-09-04.18:27:53.842>
created_at = <Date 2011-07-25.04:49:50.434>
labels = ['type-bug', 'docs']
title = 'sys.modules doc entry should reflect restrictions'
updated_at = <Date 2017-09-04.18:27:53.841>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2017-09-04.18:27:53.841>
actor = 'eric.snow'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-09-04.18:27:53.842>
closer = 'eric.snow'
components = ['Documentation']
creation = <Date 2011-07-25.04:49:50.434>
creator = 'eric.snow'
dependencies = []
files = ['22760']
hgrepos = []
issue_num = 12633
keywords = ['patch']
message_count = 9.0
messages = ['141068', '141075', '141123', '141399', '158636', '158639', '182259', '191834', '301239']
nosy_count = 4.0
nosy_names = ['ncoghlan', 'eric.araujo', 'docs@python', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = '28411'
type = 'behavior'
url = 'https://bugs.python.org/issue12633'
versions = ['Python 3.3']

[ericsnowcurrently]

The sys.modules dict is a special object. It is the only variable of the CPython interpreter object that is exposed in the sys module[1]. Everything else in sys lives in the module. However, the modules dict lives in the interpreter object and is bound to the sys module separately. No other variable of the interpreter object gets this treatment.

This situation sets up an unexpected behavior for sys.modules. There are many places, mostly in Python/import.c, where the modules dict gets used and not by pulling from sys.modules. These places use interp->modules directly[2]. So if sys.modules is re-bound, the imp module is using/reporting an out of sync modules dict.

One could argue that re-binding a module global is risky and should be avoided. I agree. Here is the use case that prompted me to march ahead anyway:

class BaseTest(TestCase):
    @classmethod
    def setUpClass(cls):
        cls.sysmodules = sys.modules
        sys.modules = sys.modules.copy()
    @classmethod
    def tearDownClass(cls):
        sys.modules = cls.sysmodules

I was writing some import related tests and wanted sys.modules to be returned to its initial state after each test. I realise that Lib/test/support.py provides CleanImport and others address this, but you have to provide the module names to clean up. This is an unfortunate hassle sometimes when several layers of imports happen during the import of the module you care about.

So the result was an exception when I tried importing an extension module, like "_sqlite3". This is because in importdl.h the new module is added to the dict returned by PyImport_GetModuleDict(), not to the one at sys.modules.

For now I am doing the following to get the same effect:

class BaseTest(TestCase):
    @classmethod
    def setUpClass(cls):
        cls.sysmodules = sys.modules.copy()
    @classmethod
    def tearDownClass(cls):
        for name in sys.modules:
            del sys.modules[name]
        for name in cls.sysmodules:
            sys.modules[name] = cls.sysmodules[name]

However, this is less efficient, sort of. I expect that the current direct use of interp->modules in the CPython code is [much?] more efficient than PySys_GetObject("modules") calls.

Proposal

In light of all this I recommend that either use of interp->modules be replaced by PySys_GetObject("modules") calls, or the sys module documentation[3] be updated to make clear that sys.modules should not be re-bound (in a CPython implementation detail note). I'm guessing that the first option is right out. The documentation addition would be just right.

[1] variables of the interpreter object found by grepping "interp->" in the CPython source:
modules
modules_by_index
next
codec_search_path
codec_search_cache
codec_error_registry
codecs_initialized
fscodec_initialized
modules_reloading
builtins
sysdict
tstate_head
tscdump
dlopenflags
[2] see PyImport_GetModuleDict() in Python/import.c
[3] http://docs.python.org/dev/library/sys.html#sys.modules

[ncoghlan]

+1 for making this limitation explicit. See the caveat on locals() [1] for an example of how to note this kind of restriction.

[1] http://docs.python.org/dev/library/functions.html#locals

[ericsnowcurrently]

Would an implementation detail note be inappropriate here? I only ask because it looks like the imp module's use of interp->modules is implementation specific.

Here's a patch for Doc/library/sys.rst that adds the note.

[merwok]

The note’s spirit is good, but I think something more concise would do.

Side note: Please don’t mix up unrelated cosmetic changes in your diffs.

[ericsnowcurrently]

The original motivator: http://mail.python.org/pipermail/python-dev/2011-July/112497.html

[ericsnowcurrently]

also, bpo-14615 is related to making sys.modules authoritative.

[ericsnowcurrently]

One proposal would lead to the sys module growing descriptors:

http://mail.python.org/pipermail/python-ideas/2013-January/019075.html

In that case, sys.modules could update the underlying interp->modules.

[ericsnowcurrently]

bpo-17953 addressed part of this.

[ericsnowcurrently]

We're dropping PyInterpreterState.modules (bpo-28411).