3.2's sysconfig doesn't work with virtualenv

[srid]

BPO	10743
Nosy	@warsaw, @birkenfeld, @pitrou, @tarekziade, @ned-deily, @merwok

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tarekziade'
closed_at = <Date 2010-12-23.18:27:07.970>
created_at = <Date 2010-12-20.22:06:41.506>
labels = ['OS-mac', 'invalid', 'type-bug', 'library', 'release-blocker']
title = "3.2's sysconfig doesn't work with virtualenv"
updated_at = <Date 2011-02-25.18:36:42.400>
user = 'https://bugs.python.org/srid'

bugs.python.org fields:

activity = <Date 2011-02-25.18:36:42.400>
actor = 'palm.kevin'
assignee = 'tarek'
closed = True
closed_date = <Date 2010-12-23.18:27:07.970>
closer = 'srid'
components = ['Distutils', 'macOS']
creation = <Date 2010-12-20.22:06:41.506>
creator = 'srid'
dependencies = []
files = []
hgrepos = []
issue_num = 10743
keywords = []
message_count = 14.0
messages = ['124405', '124412', '124413', '124439', '124441', '124442', '124546', '124562', '129361', '129367', '129370', '129373', '129388', '129399']
nosy_count = 8.0
nosy_names = ['barry', 'georg.brandl', 'pitrou', 'tarek', 'ned.deily', 'eric.araujo', 'srid', 'palm.kevin']
pr_nums = []
priority = 'release blocker'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue10743'
versions = ['Python 3.2']

[srid]

From http://code.google.com/p/virtualenv5/issues/detail?id=6 - it seems that the sysconfig module is looking for Makefile in wrong directory, while ideally it must be looking into the base Python install.

>> import sysconfig; sysconfig.get_paths('purelib')
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: '/tmp/e/lib/python3.2/config-3.2m/Makefile'

[ned-deily]

That does seem to be a regression since distutils.sysconfig works correctly in a virtualenv:

$ python3.2 -c 'import distutils.sysconfig;print(distutils.sysconfig.get_makefile_filename())'
/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/config-3.2m/Makefile
$ python3.2 -c 'import sysconfig;print(sysconfig.get_makefile_filename())'
/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/config-3.2m/Makefile

but in a virtualenv:

$ source t/bin/activate
$ python3.2 -c 'import distutils.sysconfig;print(distutils.sysconfig.get_makefile_filename())'
/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/config-3.2m/Makefile
$ python3.2 -c 'import sysconfig;print(sysconfig.get_makefile_filename())'
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: '/Users/nad/t/lib/python3.2/config-3.2m/Makefile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 322, in get_makefile_filename
    return os.path.join(get_path('stdlib'),
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 451, in get_path
    return get_paths(scheme, vars, expand)[name]
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 442, in get_paths
    return _expand_vars(scheme, vars)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 168, in _expand_vars
    _extend_dict(vars, get_config_vars())
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 487, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/sysconfig.py", line 337, in _init_posix
    raise IOError(msg)

BTW, your example will fail in any case since get_paths takes a scheme name; "purelib" is a path name.

[ned-deily]

Setting to deferred blocker for 3.2 release manager evaluation.

[pitrou]

Agreed it should probably be a release blocker. I guess it has to do with (not) following symlinks, right?

[birkenfeld]

OK, let's make this one. Tarek to the rescue!

[tarekziade]

Will do tonight

[tarekziade]

I have investigated the problem and it turns out virtualenv patches distutils.sysconfig behavior by adding to the sys module a "real_prefix" attribute that points to the global Python install and is used instead of sys.prefix that points to the virtualenv local install, when some distutils.sysconfig APIs are used.

The fix is to change virtualenv so it patches sysconfig the same way it does for distutils.sysconfig.

To simulate this patch, just change how _EXEC_PREFIX and _PREFIX are set in sysconfig, by setting them to sys.real_prefix instead of sys.prefix/sys.exec_prefix when using virtualenv.

I suggest to Georg that this bug gets resolved to "invalid" and that we move it to the virtualenv bitbucket tracker.

[srid]

Sounds good, but this doesn't belong to the virtualenv bug tracker (virtualenv does even support Python 3). Instead, it belongs to the virtualenv5 tracker: http://code.google.com/p/virtualenv5/issues/detail?id=6

[palmkevin]

I think that this issue needs to be reopened... since it never has been resolved...
I just downloaded the new version of Python 3.2 and tried to compile, install and use it on Redhat Linux.

Installation went fine (configure with --prefix=/usr/mysoft/python32)...

But when using Python I get the following error message:
Traceback (most recent call last):
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: 'lib/python3.2/config-3.2m/Makefile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 529, in <module>
    main()
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 517, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 263, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 238, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/mysoft/python32/lib/python3.2/site.py", line 228, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 590, in get_config_var
    return get_config_vars().get(name)
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 487, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/mysoft/python32/lib/python3.2/sysconfig.py", line 337, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open lib/python3.2/config-3.2m/Makefile (No such file or directory)

... I never heard about virtualenv before and I don't know what is the impact of that library, but I guess a just released version of Python should work out of the box?!? (Final 3.2 release)
(I was using Python 3.1.3 on the same machine without any hitch...)

[palmkevin]

Please find here a small C app embedding python that shows how to reproduce the problem (It turned out that the problem is caused by the new method "Py_SetPath"):

#include "Python.h"

main(int argc, char **argv)
{
  Py_SetPath(Py_GetPath());
  printf("Init\n");
  Py_Initialize();
  printf("-- END\n");
}

--> Python crashes on Py_Initialize... only if the Py_SetPath is executed...

[pitrou]

Le vendredi 25 février 2011 à 13:58 +0000, Palm Kevin a écrit :

Palm Kevin <kevin.palm@labsolution.lu> added the comment:

Please find here a small C app embedding python that shows how to
reproduce the problem (It turned out that the problem is caused by the
new method "Py_SetPath"):

Can you open a separate issue?
Also, please include your error message.

[palmkevin]

Done: bpo-11320

[warsaw]

On Feb 25, 2011, at 01:16 PM, Palm Kevin wrote:

I think that this issue needs to be reopened... since it never has been
resolved... I just downloaded the new version of Python 3.2 and tried to
compile, install and use it on Redhat Linux.

Installation went fine (configure with --prefix=/usr/mysoft/python32)...

On Ubuntu 10.10, I built Python 3.2 from the svn head with
--prefix=/tmp/prefix, did a 'make install' and then ran
/tmp/prefix/bin/python. It worked fine and I did not get the exception.

[palmkevin]

@barry: Yes python runs correctly.
The problem arises when you embed python in a C application when calling the Py_SetPath-method (see the follow up issue bpo-11320)