sys.path[0] Is Set Differently From the Rest of sys.path

[ericsnowcurrently]

Feature or enhancement

Currently sys.path[0] is set by pymain_run_python() (in Modules/main.c). This happens after pymain_init(), which initializes the runtime, including the rest of sys.path (via getpath.py and site.py). This makes it harder to reason about and introduces extra complexity for subinterpreters. (See gh-109793 and gh-109794.)

We should consider calculating sys.path[0] and setting it to its own PyConfig field via getpath.py, when the rest of the base sys.path is calculated. We may need a later check to verify that there is a matching importer, as pymain_run_python() does. (FWIW, it isn't clear that there's any value to storing the sys.path[0] value on the global _PyPathConfig.)

Also, we currently wait to actually set sys.path[0] (for the main interpreter) until after the readline/rlcompleter modules are imported in pymain_run_python(). We'd need to factor that in.

CC @zooba, @vstinner, @ncoghlan

Linked PRs

• gh-109853: Fix sys.path[0] For Subinterpreters #109994
• [3.12] gh-109853: Fix sys.path[0] For Subinterpreters (gh-109994) #110701

[vstinner]

We should consider calculating sys.path[0] and setting it to its own PyConfig field via getpath.py

There are two cases:

• (A) PyConfig.run_filename is set: pymain_get_importer() computes sys.path[0]
• (B) Otherwise, _PyPathConfig_ComputeSysPath0() computes sys.path[0]

For the case (A), do you want to execute PyImport_GetImporter() twice? Once in Py_InitializeFromConfig(), then again in pymain_run_python()? It's to decide if pymain_run_module() or pymain_run_file() should be used.

For case (B), this code path can be easily moved to Py_InitializeFromConfig(). When I designed and implemented PyConfig, I tried to minimize changes. But apparently, now the dust has settled, and we can go further :-)

[zooba]

I think we should move most of the default sys.path calculation into python.c, including the running of getpath.py (we'd need to expose the ability to create and then close a runtime that can't import anything).

If we're able to fully initialise the search path using only our public APIs, we'll have a much better interface for embedders to use.

[vstinner]

I would love that sys.path would be fully initialized before the site module is loaded. Currently, sys.path is still modified by the site module in many ways, and so python -S gives a different sys.path :-(

site changes:

• Make paths absolute (why not doing that earlier?)
• Add user site directory (is it complicated to move the logic to getpath?)

[zooba]

I expect most of the site module can move into getpath. Venv and pth sure can (though we'd have to defer code execution in pth files until after initialization finishes). Some of the interactive mode features probably can't, but I'd also like to treat those as something specific to python.c and separate from libpython (i.e. part of the Python program not the Python interpreter).

[encukou]

This adds PyConfig.sys_path_0 as public API.
Should we add some documentation for it, or mark it internal (add an underscore)?

[zooba]

Need @ericsnowcurrently to confirm, but I suspect marking it internal is better. When the calculation gets refactored into getpath.py then there shouldn't be any need to store it separately.

[ericsnowcurrently]

This adds PyConfig.sys_path_0 as public API. Should we add some documentation for it, or mark it internal (add an underscore)?

We should mark it as internal at least for now. We'd need to sort out the complexity I described above before this would become meaningful config.

[ncoghlan]

The sys.path[0] initialisation semantics are even worse than @ericsnowcurrently describes, since runpy may mutate the value if it gets invoked via -m or path entry execution.

There's an intrinsic problem here in that sys.path[0] is not semantically identical to other sys.path entries (it can be set from a much wider variety of sources, including being dropped entirely when running in isolated mode), but once the desired value is figured out, we do want it to be treated the same as any other entry for module import purposes (hence it being in the list rather than stored somewhere else).

[zooba]

I wonder if we can make from . import <mod> work from __main__ easily so there's a way to transition towards -P (no sys.path[0] by default)? Or if that's even worth attempting?