building modules from 3.8.0rc1 fails in a venv when system 3.8 is present

[pmp-p]

BPO	38393
Nosy	@vstinner, @pmp-p

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-10-07.13:27:24.317>
labels = ['3.8', 'build', '3.7']
title = 'building modules from 3.8.0rc1 fails in a venv when system 3.8 is present'
updated_at = <Date 2019-10-07.14:04:56.889>
user = 'https://github.com/pmp-p'

bugs.python.org fields:

activity = <Date 2019-10-07.14:04:56.889>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2019-10-07.13:27:24.317>
creator = 'pmpp'
dependencies = []
files = []
hgrepos = []
issue_num = 38393
keywords = []
message_count = 3.0
messages = ['354095', '354096', '354098']
nosy_count = 2.0
nosy_names = ['vstinner', 'pmpp']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue38393'
versions = ['Python 3.7', 'Python 3.8']

[pmp-p]

when trying to build rc1 from a b4 venv to prepare a python host for cross compilation on ubuntu xenial flavour x64

i got :

cd /tmp
python3.7 -m venv testenv
cd testenv/
. bin/activate
(testenv) /tmp/testenv $ tar xf /tmp/Python-3.8.0rc1.tar.xz && cd Python-3.8.0rc1

CC=clang ./configure --prefix=/tmp/python3.host --with-system-ffi --disable-ipv6 --without-ensurepip --with-c-locale-coercion --disable-shared && make

./python -E ./setup.py build
step will fail for the reason i suspect is srcdir is taken from system python3.8 if installed ( /usr/local in that case )

$ ./python -E 
Python 3.8.0rc1 (default, Oct  7 2019, 15:16:07) 
[Clang 6.0.0 (tags/RELEASE_600/final)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('srcdir')
'/usr/local/lib/python3.8/config-3.8-x86_64-linux-gnu'
>>>

[vstinner]

It's not a Python 3.8 regression. Python 3.7 has the same behavior.

cd $HOME
python3.7 -m venv testenv
cd testenv/
. bin/activate

wget https://www.python.org/ftp/python/3.8.0/Python-3.8.0rc1.tar.xz
tar -xf Python-3.8.0rc1.tar.xz && cd Python-3.8.0rc1
CC=clang ./configure --prefix=/tmp/python3.host --with-system-ffi --disable-ipv6 --without-ensurepip --with-c-locale-coercion --disable-shared && make

Python 3.8 error:

"""
$ make
./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = './python'
isolated = 0
environment = 0
user site = 1
import site = 0
sys._base_executable = '/home/vstinner/testenv/Python-3.8.0rc1/python'
sys.base_prefix = '/tmp/python3.host'
sys.base_exec_prefix = '/tmp/python3.host'
sys.executable = '/home/vstinner/testenv/Python-3.8.0rc1/python'
sys.prefix = '/tmp/python3.host'
sys.exec_prefix = '/tmp/python3.host'
sys.path = [
'/tmp/python3.host/lib/python38.zip',
'/tmp/python3.host/lib/python3.8',
'/tmp/python3.host/lib/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff6a75f6740 (most recent call first):
<no Python frame>
generate-posix-vars failed
make: *** [Makefile:592: pybuilddir.txt] Error 1
"""

cd ..
wget https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tar.xz
tar -xf Python-3.7.4.tar.xz
cd Python-3.7.4/
CC=clang ./configure --prefix=/tmp/python3.host --with-system-ffi --disable-ipv6 --without-ensurepip --with-c-locale-coercion --disable-shared

Python 3.7 error:

"""
$ make
./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
echo "generate-posix-vars failed" ; \
rm -f ./pybuilddir.txt ; \
exit 1 ; \
fi
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: initfsencoding: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f5a77f5a740 (most recent call first):
/bin/sh : ligne 5 : 19592 Aborted (core dumped)./python -E -S -m sysconfig --generate-posix-vars
generate-posix-vars failed
make: *** [Makefile:606: pybuilddir.txt] Error 1
"""

[vstinner]

The problem is that Python looks for pyvenv.cfg in the program directory or in its parent directory. For example:

/tmp/testenv/Python-3.8.0rc1/python looks for /tmp/testenv/Python-3.8.0rc1/pyvenv.cfg (doesn't exist) and
/tmp/testenv/Python-3.8.0rc1/pyvenv.cfg (exists). On my system, pyvenv.cfg contains:
---
home = /usr/bin
include-system-site-packages = false
version = 3.7.4
---

We should add an option to ignore pyvenv.cfg, or modify an existing option like -I to ignore pyvenv.cfg.