sysconfig.get_config_vars('srcdir') fails in specific cases

[tarekziade]

BPO	12141
Nosy	@pitrou, @tarekziade, @ned-deily, @merwok, @bitdancer
Files	issue12141_3x.patch: 3.3 version for packaging and distutils issue12132_backport_32.patch: 3.2 backport of distutils part of issue12132 issue12141_32.patch: 3.2 version for distutils issue12141_27.patch: 2.7 version for distutils

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tarekziade'
closed_at = <Date 2011-06-29.00:21:51.621>
created_at = <Date 2011-05-21.22:32:30.581>
labels = ['library']
title = "sysconfig.get_config_vars('srcdir') fails in specific cases"
updated_at = <Date 2011-09-04.06:41:51.875>
user = 'https://github.com/tarekziade'

bugs.python.org fields:

activity = <Date 2011-09-04.06:41:51.875>
actor = 'python-dev'
assignee = 'tarek'
closed = True
closed_date = <Date 2011-06-29.00:21:51.621>
closer = 'ned.deily'
components = ['Distutils', 'Library (Lib)', 'Distutils2']
creation = <Date 2011-05-21.22:32:30.581>
creator = 'tarek'
dependencies = []
files = ['22346', '22347', '22348', '22349']
hgrepos = []
issue_num = 12141
keywords = ['patch']
message_count = 20.0
messages = ['136483', '136484', '136486', '136487', '136489', '136493', '136494', '136568', '138269', '138283', '138284', '138319', '138609', '139001', '139012', '139349', '139350', '140438', '142548', '143478']
nosy_count = 8.0
nosy_names = ['pitrou', 'nadeem.vawda', 'tarek', 'ned.deily', 'eric.araujo', 'r.david.murray', 'alexis', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue12141'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[tarekziade]

this test module looks for sysconfig.get_config_var('srcdir') which in turns uses the sys,executable path.

multiprocess seems to change it in every process, leading to the errors.

To reproduce:

./python Lib/test/regrtest.py -j2 -v test_packaging

A workaround is to skip the test in case the file is not found, but we need to fix it because it boils down to sysconfig being broken in multiprocess

[pitrou]

What's the difference with bpo-12132?

[tarekziade]

Oops. It's a duplicate. Keeping this one since the problem was narrowed to multiprocessing/sys,executable and sysconfig

[pitrou]

Oops. It's a duplicate. Keeping this one since the problem was
narrowed to multiprocessing/sys,executable and sysconfig

I don't think it has anything to do with sys.executable. As the other
issue says, it seems the test (or packaging/sysconfig itself) is simply
unable to locate the root of the current checkout (which should be
trivial using __file__).

[tarekziade]

sysconfig is looking for the source dir when

  sysconfig.get_config_var('srcdir')

is called.

And this is done like this:

if sys.executable:
    _PROJECT_BASE = os.path.dirname(_safe_realpath(sys.executable))
else:
    # sys.executable can be empty if argv[0] has been changed and Python is
    # unable to retrieve the real program name
    _PROJECT_BASE = _safe_realpath(os.getcwd())

Because sys.executable (argv[0] in fact) is not filled when you call multiprocess vvia the -j option.

So yeah, this has to do with sys.executable

[pitrou]

Because sys.executable (argv[0] in fact) is not filled when you call
multiprocess vvia the -j option.

sys.executable is set perfectly well when running regrtest in
multiprocess mode, otherwise many other tests would fail.

Try:
$ grep "sys.executable" Lib/test/*
Lib/test/regrtest.py:585: base_cmd = [sys.executable] + opt_args + ['-m', 'test.regrtest']
Lib/test/script_helper.py:20: cmd_line = [sys.executable]
Lib/test/script_helper.py:60: cmd_line = [sys.executable, '-E']
Lib/test/test_base64.py:231: args = (sys.executable, '-m', 'base64') + args
Lib/test/test_capi.py:42: p = subprocess.Popen([sys.executable, "-c",
[etc.]

Again, it seems sysconfig simply has a bug which you can reproduce on
the command line:

$ pwd
/home/antoine/cpython/default/build
$ ../python -c "import sys, sysconfig; print(sys.executable,
sysconfig.get_config_var('srcdir'))"
/home/antoine/cpython/default/build/../python /home/antoine/cpython/default/build

[ned-deily]

The test should be changed anyway to avoid the dependency on "srcdir", whose value has no meaning when the tests are not run from a simple in-source-directory build. The test could create its own temp copy of xxmodule.c.

[tarekziade]

@ned: right. done, and fixes bpo-12132

[ned-deily]

Here are patches to install a copy of xxmodule.c in the distutils tests directory (for 3.3, 3.2, and 2.7) and a copy in packaging tests for 3.3. With them in place, test_build_ext/test_command_build_ext now executes when the tests are run from an installed Python where no source directory is available and each test case run copies the c file into its temp dir.

Currently, those test are being silently skipped in the installed case or, worse, picking up the wrong version of xxmodule.c depending on what sysconfig.get_config_vars('srcdir') returns (this happened to me when I reused a source directory name in some builds). For 3.2, a patch is included to backport the Distutils part of the fix for bpo-12132.