Add sys.stdlib_module_names: list of stdlib module names (Python and extension modules)

[vstinner]

BPO	42955
Nosy	@ronaldoussoren, @vstinner, @aroberge, @serhiy-storchaka, @SylvainDe, @corona10, @shihai1991
PRs	bpo-42955: Add sys.modules_names #24238 bpo-42923: _Py_DumpExtensionModules() ignores stdlib ext #24254 bpo-42955: Add Python/module_names.h #24258 bpo-42955: Fix sys.module_names doc #24329 bpo-42955: Rename module_names to sys.stdlib_module_names #24332 bpo-42955: Remove sub-packages from sys.stdlib_module_names #24353 bpo-42955: Add _overlapped to sys.stdlib_module_names #25122

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-25.12:30:06.176>
created_at = <Date 2021-01-18.10:08:34.048>
labels = ['library', '3.10']
title = 'Add sys.stdlib_module_names: list of stdlib module names (Python and extension modules)'
updated_at = <Date 2021-04-01.00:28:36.061>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-04-01.00:28:36.061>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-25.12:30:06.176>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2021-01-18.10:08:34.048>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 42955
keywords = ['patch']
message_count = 33.0
messages = ['385180', '385181', '385186', '385194', '385197', '385200', '385201', '385208', '385250', '385251', '385252', '385253', '385254', '385255', '385270', '385272', '385273', '385300', '385310', '385311', '385312', '385317', '385350', '385364', '385493', '385623', '385624', '385625', '385670', '385671', '385817', '388419', '389942']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'vstinner', 'aroberge', 'serhiy.storchaka', 'SylvainDe', 'corona10', 'shihai1991']
pr_nums = ['24238', '24254', '24258', '24329', '24332', '24353', '25122']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42955'
versions = ['Python 3.10']

[vstinner]

Some use cases require to know if a module comes from the stdlib or not. For example, I would like to only dump extension modules which don't come from the stdlib in bpo-42923: "Py_FatalError(): dump the list of extension modules".

Stdlib modules are special. For example, the maintenance and updates are connected to the Python lifecycle. Stdlib modules cannot be updated with "pip install --upgrade". They are shipped with the system ("system" Python). They are usually "read only": on Unix, only the root user can write into /usr directory where the stdlib is installed, whereas modules installed with "pip install --user" can be modified by the current user.

There is a third party project on PyPI which contains the list of stdlib modules:
https://pypi.org/project/stdlib-list/

There is already sys.builtin_module_names:
"A tuple of strings giving the names of all modules that are compiled into this Python interpreter."
https://docs.python.org/dev/library/sys.html#sys.builtin_module_names

I propose to add a similar sys.module_names tuple of strings (module names).

There are different constraints:

• If we add a public sys attribute, users will likely expect the list to be (1) exhaustive (2) correct
• Some extensions are not built if there are missing dependencies. Dependencies are checked after Python "core" (the sys module) is built.
• Some extensions are not available on some platforms.
• This list should be maintained.

Should we only list top level packages, or also submodules? For example, only list "asyncio", or list the 31 submodules (asyncio.base_events, asyncio.futures, ...)? Maybe it can be decided on a case by case basis. For example, I consider that "os.path" is a stdlib module, even it's just an alias to "posixpath" or "ntpath" depending on the platform.

I propose to include all extensions in the list, even if they are not built/available on some platforms. For example, "winsound" would also be listed on Linux, even if the extension is specific to Windows.

I also propose to include stdlib module names even if they are overridden at runtime using PYTHONPATH with a different implementation. For example, "asyncio" would be in the list, even if an user creates "asyncio.py" file. The list would not depend on sys.path.

--

Another option is to add an attribute to modules to mark them as coming from the stdlib. The API would be an attribute: module.__stdlib__ (bool).

The attribute could be set directly in the module code. For example, add "__stdlib__ = True" in Python modules. Similar idea for C extension modules.

Or the attribute could be set after importing the module, in the import site. But we don't control how stdlib modules are imported.

--

For the specific case of bpo-42923, another option is to use a list of stdlib paths, and check module.__file__ to check if a module is a stdlib module, and also use sys.builtin_module_names. And so don't add any API.

[vstinner]

On Fedora 33, the stdlib lives in two main directories:

• /usr/lib64/python3.9: Python modules
• /usr/lib64/python3.9/lib-dynload: C extension modules

Example:

>>> import os.path
>>> os.path.dirname(os.__file__) # Python
'/usr/lib64/python3.9'
>>> os.path.dirname(_asyncio.__file__)
'/usr/lib64/python3.9/lib-dynload'

The Python stdlib path can be retrieved with:

>>> import sysconfig; sysconfig.get_paths()['stdlib']
'/usr/lib64/python3.9'

But I'm not sure how to retrieve /usr/lib64/python3.9/lib-dynload path. "platstdlib" is not what I expect:

>>> import sysconfig; sysconfig.get_paths()['platstdlib']
'/usr/lib64/python3.9'

I found DESTDIR in Makefile:

>>> sysconfig.get_config_var('DESTSHARED')
'/usr/lib64/python3.9/lib-dynload'

[serhiy-storchaka]

There is no technical difference between stdlib modules and other modules. Stdlib modules are only special in context of copyright and responsibility.
What if set __author__ = 'PSF' in every stdib module? Or other attributes which would allow to group modules by origin. Perhaps other authors would use that feature too.

[vstinner]

I wrote PR 24238 which implements an hardcoded list of stdlib module names. The PR uses the CI to ensure that the list always remain up to date. If tomorrow a new module is added, a CI job fails ("file not up to date").

What if set __author__ = 'PSF' in every stdib module? Or other attributes which would allow to group modules by origin. Perhaps other authors would use that feature too.

Hum, that would go against my bpo-42923 use case.

Also for bpo-42923, I would prefer a tuple of strings ;-)

[vstinner]

Use cases for sys.module_names:

• When computing dependencies of a project, ignore modules which are part of the stdlib: Find all libraries required by a given package jackmaney/pypt#3

• Trace the execution of third party code, but ignore stdlib, use --ignore-module option of trace: https://stackoverflow.com/questions/6463918/how-can-i-get-a-list-of-all-the-python-standard-library-modules

• When reformatting a Python source file, group imports of stdlib modules. The isort module contains the list of stdlib modules, one list py Python version: https://github.com/PyCQA/isort/tree/develop/isort/stdlibs which is generated from the online Python documentation.

---

The isort uses the following script to generate the list of stdlib modules:
https://github.com/PyCQA/isort/blob/develop/scripts/mkstdlibs.py

The script uses sphinx.ext.intersphinx.fetch_inventory(...)["py:module"]. This API uses objects.inv from the online Python documentation. Example of Python 3.9:

https://docs.python.org/3.9/objects.inv

On the "dev" version, mkstdlibs.py lists 211 modules.

[vstinner]

(...) This API uses objects.inv from the online Python documentation.

By the way, there is also this documentation page listing Python stdlib modules:
https://docs.python.org/dev/py-modindex.html

[vstinner]

Another way to gather the list of Python modules: pydoc help("modules") uses pkgutil.walk_packages().

pydoc uses ModuleScanner which uses sys.builtin_module_names and pkgutil.walk_packages().

pkgutil.walk_packages() calls pkgutil.iter_modules() which iterates on sys.meta_path and sys.path, and calls iter_modules() on each importer.

For FileImporter, it iterates on os.listdir() on the importer path.

For zipimporter, it iterates on zipimport._zip_directory_cache[importer.archive].

[vstinner]

Note for myself: regen-keyword should be included in "make regen-all".

[ronaldoussoren]

A list of stdlib modules/extensions is IMHO problematic for maintenance, esp. if you consider that not all modules/extensions are installed on all systems (both because dependencies aren't present and because packagers have decided to unbundle parts of the stdlib).

Wouldn't it be sufficient to somehow mark the stdlib entries on sys.path? Although that might give misleading answers with tools like pyinstaller/py2exe/py2app that package an application and its dependencies into a single zipfile.

[vstinner]

Ronald Oussoren:

A list of stdlib modules/extensions is IMHO problematic for maintenance, esp. if you consider that not all modules/extensions are installed on all systems (both because dependencies aren't present and because packagers have decided to unbundle parts of the stdlib).

My PR 24238 adds a list of module names (tuple of str): sys.module_names. It includes modules which are not available on the platform. For example, "winsound" is listed on Linux but not available, and "tkinter" is listed even if the extension could not be built (missing dependency).

I tried to make it clear in the sys.module_names documentation (see my PR).

Making the list conditional depending if the module is built or not is causing different issues. For example, for the "isort" (sort and group imports) use case, you want to know on Linux if "import winsound" is a stdlib import or a third party import (same for Linux-only modules on Windows). Moreover, there is a practical issue: extension modules are only built by setup.py *after* the sys module is built. I don't want to rebuild the sys module if building an extension failed.

Even if a module is available (listed in sys.module_names and the file is present on disk), it doesn't mean that "it works". For example, "import multiprocessing" fails if there is no working lock implementation. Other modules have similar issues.

Wouldn't it be sufficient to somehow mark the stdlib entries on sys.path? Although that might give misleading answers with tools like pyinstaller/py2exe/py2app that package an application and its dependencies into a single zipfile.

Having to actually import modules to check if it's a stdlib module or not is not convenient. Many stdlib modules have side effects on import. For example, "import antigravity" opens a web browser. An import can open files, spawn threads, run programs, etc.

[ronaldoussoren]

> Wouldn't it be sufficient to somehow mark the stdlib entries on sys.path? Although that might give misleading answers with tools like pyinstaller/py2exe/py2app that package an application and its dependencies into a single zipfile.

Having to actually import modules to check if it's a stdlib module or not is not convenient. Many stdlib modules have side effects on import. For example, "import antigravity" opens a web browser. An import can open files, spawn threads, run programs, etc.

You wouldn't necessarily have to import a module to test, this is something that could be added to importlib. One (poorly thought out) option is to add sys._stdlib_path with the subsection of sys.path that contains the stdlib, and a function in importlib that returns if a spec is for a stdlib module.

The disadvantage of this, or for the most part anything but your initial proposal, is that might not be save to use a function in importlib in Py_FatalError.

[vstinner]

Myself:

Having to actually import modules to check if it's a stdlib module or not is not convenient. Many stdlib modules have side effects on import.

The "trace" usecase needs an exhaustive list of all module names. It is even less convenient to have to list all Python modules available on the system only to check modules coming from the stdlib.

Ronald:

You wouldn't necessarily have to import a module to test, this is something that could be added to importlib. One (poorly thought out) option is to add sys._stdlib_path with the subsection of sys.path that contains the stdlib, and a function in importlib that returns if a spec is for a stdlib module.

I'm not sure how it would work. I listed different cases which have different constraints:

• From a module name, check if it's part of the stdlib or not
• From a module object, check if it's part of the stdlib or not

For the test on the module name, how would it work with sys._stdlib_path? Should you import the module and then check if its path comes from sys._stdlib_path?

Ronald:

The disadvantage of this, or for the most part anything but your initial proposal, is that might not be save to use a function in importlib in Py_FatalError.

PR 24254 is a working implementation of my use case: only list third party extension modules on a Python fatal error. It relies on PR 24238 sys.module_names list. The implementation works when called from a signal handler (when faulthandler catch fatal signals like SIGSEGV), it avoids memory allocations on the heap (one of the limits of a signal handler).

[ronaldoussoren]

On 19 Jan 2021, at 12:30, STINNER Victor <report@bugs.python.org> wrote:

Ronald:
> You wouldn't necessarily have to import a module to test, this is something that could be added to importlib. One (poorly thought out) option is to add sys._stdlib_path with the subsection of sys.path that contains the stdlib, and a function in importlib that returns if a spec is for a stdlib module.

I'm not sure how it would work. I listed different cases which have different constraints:

• From a module name, check if it's part of the stdlib or not
• From a module object, check if it's part of the stdlib or not

For the test on the module name, how would it work with sys._stdlib_path? Should you import the module and then check if its path comes from sys._stdlib_path?

For a module name use importlib.util.find_spec() to locate the module (or toplevel package if this is a module in package). The new importlib function could then use the spec and sys._stdlib_path to check if the spec is one for a stdlib module. This is pretty handwavy, but I do something similar in py2app (but completely based on paths calculated outside of the import machinery).

For a module object you can extract the spec from the object and use the same function.

Ronald:
> The disadvantage of this, or for the most part anything but your initial proposal, is that might not be save to use a function in importlib in Py_FatalError.

PR 24254 is a working implementation of my use case: only list third party extension modules on a Python fatal error. It relies on PR 24238 sys.module_names list. The implementation works when called from a signal handler (when faulthandler catch fatal signals like SIGSEGV), it avoids memory allocations on the heap (one of the limits of a signal handler).

I think we agree on that point: my counter proposal won’t work in the faulthandler scenario, and may be problematic in the Py_FatalError case as well.

Ronald