sys.path entries normalization in site.py doesn't follow POSIX symlink behaviour

[jpoiret]

BPO	46653
Nosy	@jpoiret

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-02-05.22:20:58.875>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.11', '3.7', 'library']
title = "sys.path entries normalization in site.py doesn't follow POSIX symlink behaviour"
updated_at = <Date 2022-02-05.22:20:58.875>
user = 'https://github.com/jpoiret'

bugs.python.org fields:

activity = <Date 2022-02-05.22:20:58.875>
actor = 'jpoiret'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2022-02-05.22:20:58.875>
creator = 'jpoiret'
dependencies = []
files = []
hgrepos = []
issue_num = 46653
keywords = []
message_count = 1.0
messages = ['412600']
nosy_count = 1.0
nosy_names = ['jpoiret']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46653'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10', 'Python 3.11']

[jpoiret]

Whenever sys.prefix contains a symlink followed by a '..', the corresponding part of sys.path entries will not refer to the parent directory of the directory pointed to by the symlink, but rather to the directory in which the symlink is. Thus, it will be impossible to import standard Python modules installed at sys.prefix, among other things.

Here is an example:
Suppose you have installed Python with prefix /usr. Create a symlink /tmp/symlink pointing to /usr/lib, and launch PYTHONHOME=/tmp/symlink/.. python3. In that REPL, import warnings will fail to find the correct module, as evidenced by the value of sys.path containing entries such as /tmp/lib/python3.X/ instead of the expected /usr/lib/python3.X/.

This issue is caused by the makepath function (among others) in Lib/site.py using os.path.abspath instead of os.path.realpath, which does not follow POSIX as the documentation of os.path.normpath (used internally by abspath) suggests.

I propose replacing all four instances of abspath in Lib/site.py to realpath instead. This is a breaking change for users that relied on non-conforming symlink semantics (although that user-base might be quite small), but in my opinion Python should be expected to follow the behaviour of the platform it is running on.

This issue was raised while investigating a bug [1] in the relocatable packs feature of GNU Guix [2], which makes use of symlinks to achieve relocatability.

[1] https://issues.guix.gnu.org/53258
[2] https://guix.gnu.org/en/blog/2018/tarballs-the-ultimate-container-image-format/