Dump the Python path configuration at the first import error

[vstinner]

BPO	38236
Nosy	@ncoghlan, @vstinner, @methane, @zooba, @matrixise, @pablogsal
PRs	bpo-38236: Dump path config at first import error #16300 [3.8] bpo-38236: Dump path config at first import error (GH-16300) #16332 bpo-38236: Fix init_dump_ascii_wstr() #16333

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-09-23.14:01:37.312>
created_at = <Date 2019-09-20.16:44:55.526>
labels = ['interpreter-core', '3.8', '3.9']
title = 'Dump the Python path configuration at the first import error'
updated_at = <Date 2019-09-23.14:01:37.312>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-09-23.14:01:37.312>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-09-23.14:01:37.312>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2019-09-20.16:44:55.526>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 38236
keywords = ['patch']
message_count = 8.0
messages = ['352869', '352873', '352874', '352925', '353013', '353015', '353016', '353017']
nosy_count = 6.0
nosy_names = ['ncoghlan', 'vstinner', 'methane', 'steve.dower', 'matrixise', 'pablogsal']
pr_nums = ['16300', '16332', '16333']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38236'
versions = ['Python 3.8', 'Python 3.9']

[vstinner]

When Python path configuration is not properly configured, Python fails with an error looking like:
---
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff0065b2740 (most recent call first):
<no Python frame>
---

This error is very common and hard to debug. It can happen in various cases for various reasons:

• Python is embedded in an application
• PYTHONPATH and/or PYTHONHOME environment variable is set
• pyenv.cfg configuration for virtual environment
• Local .py files which override the standard library

A few examples:

• bpo-38234: Py_SetPath() regression in Python 3.8 which impacts PyInstaller
• https://bugs.python.org/issue35978#msg335363 virtual env
• https://bugs.python.org/issue35969#msg335249 "encodings.py" file in the current directory
• https://stackoverflow.com/questions/55657593/python-3-8-0a3-failed-to-get-the-python-codec-of-the-filesystem-encoding embedded Python
• https://askubuntu.com/questions/640010/fatal-python-error-py-initialize-unable-to-get-the-locale-encoding-syntaxe Python 3 installed by Anaconda which uses Python 2.7 stdlib because of PYTHONPATH
• Fatal Python error: Py_Initialize: Unable to get the locale encoding GNS3/gns3-gui#2267 Python embedded in "gns3"
• ImportError: No module named 'encodings' when using with virtualenv + python3 JuliaPy/PyCall.jl#410 venv
• https://techglimpse.com/error-executing-python3-5-command-solution/ another random error

Attached PR dumps the Python path configuration if importing the Python codec of the filesystem encoding fails. In the Python initialization, it's the first import which is done.

[vstinner]

Without my change:

---
vstinner@apu$ PYTHONHOME=/xxx ./python
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007fa9c32d6740 (most recent call first):
<no Python frame>
---

With my PR 16300:

---

$ PYTHONHOME=/xxx ./python 
Python path configuration:
  PYTHONHOME = '/xxx'
  PYTHONPATH = '/usr/share/qa-tools/python-modules'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys.executable = '/home/vstinner/python/master/python'
  sys.prefix = '/xxx'
  sys.exec_prefix = '/xxx'
  sys.path = [
    '/usr/share/qa-tools/python-modules',
    '/xxx/lib/python39.zip',
    '/xxx/lib/python3.9',
    '/xxx/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f9cef949740 (most recent call first):
<no Python frame>
---

[vstinner]

user site = 1
import site = 1

I'm not sure that dumping these configuration variables are useful to debug this very specific error:

"Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding"

They only have an effect on sys.path later.

[matrixise]

+1

[vstinner]

New changeset fcdb027 by Victor Stinner in branch 'master':
bpo-38236: Dump path config at first import error (GH-16300)
fcdb027

[vstinner]

New changeset 88e6447 by Victor Stinner in branch 'master':
bpo-38236: Fix init_dump_ascii_wstr() (GH-16333)
88e6447

[vstinner]

New changeset c5c6425 by Victor Stinner in branch '3.8':
bpo-38236: Dump path config at first import error (GH-16300) (GH-16332)
c5c6425

[vstinner]

I backported my change to Python 3.8.

Example with PYTHONPATH set to Python 2.7 with Python 3.8:

$ PYTHONPATH=/usr/lib64/python2.7/ ./python -c pass
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/usr/lib64/python2.7/'
  program name = './python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/home/vstinner/python/3.8/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.executable = '/home/vstinner/python/3.8/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/lib64/python2.7/',
    '/usr/local/lib/python38.zip',
    '/home/vstinner/python/3.8/Lib',
    '/home/vstinner/python/3.8/build/lib.linux-x86_64-3.8-pydebug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
  File "/usr/lib64/python2.7/encodings/__init__.py", line 123
    raise CodecRegistryError,\
                            ^
SyntaxError: invalid syntax

Current thread 0x00007fe3ade4d740 (most recent call first):
<no Python frame>