Error running locally built image with pack: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

[david-caro]

Not sure what I'm doing wrong, it might be a dummy thing, but when trying to run a locally built image (with pack) using the heroku/builder:22, I get the error:

dcaro@vulcanus$ docker run --rm wm-lol
[uWSGI] getting INI configuration from uwsgi.ini
*** Starting uWSGI 2.0.21 (64bit) on [Tue Mar 14 19:03:17 2023] ***
compiled with version: 11.3.0 on 01 January 1980 00:00:01
os: Linux-6.1.0-2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.7-1 (2023-01-18)
nodename: afb11a88e4cc
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /workspace
detected binary path: /layers/heroku_python/dependencies/bin/uwsgi
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :8000 fd 3
Python version: 3.11.2 (main, Feb  8 2023, 12:54:20) [GCC 11.3.0]
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/layers/heroku_python/dependencies/bin/uwsgi'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/app/.heroku/python/lib/python3.11'
  sys._base_executable = '/layers/heroku_python/dependencies/bin/uwsgi'
  sys.base_prefix = '/app/.heroku/python'
  sys.base_exec_prefix = '/app/.heroku/python'
  sys.platlibdir = 'lib'
  sys.executable = '/layers/heroku_python/dependencies/bin/uwsgi'
  sys.prefix = '/app/.heroku/python'
  sys.exec_prefix = '/app/.heroku/python'
  sys.path = [
    '/app/.heroku/python/lib/python311.zip',
    '/app/.heroku/python/lib/python3.11',
    '/app/.heroku/python/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f6228908440 (most recent call first):
  <no Python frame>

The build process works well:

dcaro@vulcanus$ pack build --builder heroku/builder:22 wm-lol
22: Pulling from heroku/builder
Digest: sha256:eb9486a5587e666f43695ff794a28056b18033fd2f6566c81137634087d23034
Status: Image is up to date for heroku/builder:22
22-cnb: Pulling from heroku/heroku
Digest: sha256:8ed2415c4e53df324c1af95e7286f1065a9cced71ea7e449574064a2c268d55c
Status: Image is up to date for heroku/heroku:22-cnb
latest: Pulling from buildpacksio/lifecycle
Digest: sha256:f75a04887fced3ae0504a37edb2c0d29d366511cd9ede34dbb90c5282b106e79
Status: Image is up to date for buildpacksio/lifecycle:latest
===> ANALYZING
[analyzer] Restoring data for SBOM from previous image
===> DETECTING
[detector] heroku/python   0.1.0
[detector] heroku/procfile 2.0.0
===> RESTORING
[restorer] Restoring metadata for "heroku/python:python" from app image
[restorer] Restoring metadata for "heroku/python:pip-cache" from cache
[restorer] Restoring metadata for "heroku/python:shim" from cache
[restorer] Restoring data for "heroku/python:pip-cache" from cache
[restorer] Restoring data for "heroku/python:python" from cache
[restorer] Restoring data for "heroku/python:shim" from cache
===> BUILDING
[builder]
[builder] [Determining Python version]
[builder] No Python version specified, using the current default of Python 3.11.2.
[builder] To use a different version, see: https://devcenter.heroku.com/articles/python-runtimes
[builder]
[builder] [Installing Python and packaging tools]
[builder] Using cached Python 3.11.2
[builder] Using cached pip 23.0.1, setuptools 67.5.0 and wheel 0.38.4
[builder]
[builder] [Installing dependencies using Pip]
[builder] Using cached pip download/wheel cache
[builder] Running pip install
[builder] Collecting flask
[builder]   Using cached Flask-2.2.3-py3-none-any.whl (101 kB)
[builder] Collecting werkzeug
[builder]   Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB)
[builder] Collecting uwsgi
[builder]   Using cached uWSGI-2.0.21-cp311-cp311-linux_x86_64.whl
[builder] Collecting Jinja2>=3.0
[builder]   Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
[builder] Collecting itsdangerous>=2.0
[builder]   Using cached itsdangerous-2.1.2-py3-none-any.whl (15 kB)
[builder] Collecting click>=8.0
[builder]   Using cached click-8.1.3-py3-none-any.whl (96 kB)
[builder] Collecting MarkupSafe>=2.1.1
[builder]   Using cached MarkupSafe-2.1.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (27 kB)
[builder] Installing collected packages: uwsgi, MarkupSafe, itsdangerous, click, werkzeug, Jinja2, flask
[builder] Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.2 click-8.1.3 flask-2.2.3 itsdangerous-2.1.2 uwsgi-2.0.21 werkzeug-2.2.3
[builder]
[builder] [Discovering process types]
[builder] Procfile declares types -> web
===> EXPORTING
[exporter] Adding layer 'heroku/python:dependencies'
[exporter] Reusing layer 'heroku/python:python'
[exporter] Reusing layer 'buildpacksio/lifecycle:launch.sbom'
[exporter] Adding 1/1 app layer(s)
[exporter] Reusing layer 'buildpacksio/lifecycle:launcher'
[exporter] Adding layer 'buildpacksio/lifecycle:config'
[exporter] Reusing layer 'buildpacksio/lifecycle:process-types'
[exporter] Adding label 'io.buildpacks.lifecycle.metadata'
[exporter] Adding label 'io.buildpacks.build.metadata'
[exporter] Adding label 'io.buildpacks.project.metadata'
[exporter] Setting default process type 'web'
[exporter] Saving wm-lol...
[exporter] *** Images (f36700bd8dd2):
[exporter]       wm-lol
[exporter] Adding cache layer 'heroku/python:pip-cache'
[exporter] Reusing cache layer 'heroku/python:python'
Successfully built image wm-lol

I can run it though if I set both PYTHONHOME and LD_LIBRARY_PATH to point to the python layer dirs:

docker run --rm --env LD_LIBRARY_PATH=/layers/heroku_python/python/lib/ --env PYTHONHOME=/layers/heroku_python/python/

[david-caro]

The code I'm running is https://github.com/david-caro/wm-lol (simple python app uwsgi+flask)

[david-caro]

I think this might be actually for https://github.com/heroku/buildpacks-python, will move there (the docs point here though).

[david-caro]

Moved to the right repo :)

[edmorley]

I think this might be actually for https://github.com/heroku/buildpacks-python, will move there

Yeah the reported issue is about the CNB, which lives at https://github.com/heroku/buildpacks-python rather than here.

(the docs point here though).

Which docs? Heroku doesn't yet use CNBs (CNBs are still experimental), so https://devcenter.heroku.com/articles/buildpacks#officially-supported-buildpacks linking to here is correct. I agree there is potential for confusion here though - there may be things we can do to make thing less confusing as we go through the transition from classic buildpacks to CNBs in the future.