python2.7 sysconfig broken under virtualenv #118

Closed
bukzor opened this Issue Apr 13, 2011 · 18 comments

Projects

None yet
@bukzor
bukzor commented Apr 13, 2011

This is preventing numpy from installing correctly:

$ virtualenv --version
1.6

>>> import sysconfig
>>> sysconfig.get_config_vars()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/sysconfig.py", line 444, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/lib/python2.7/sysconfig.py", line 298, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open /home/bgolemon/software/sqlalchemy/local/lib/python2.7/config/Makefile (No such file or directory)

>>> sysconfig._get_makefile_filename()
'/home/bgolemon/software/sqlalchemy/local/lib/python2.7/config/Makefile'

Quick fix:
mkdir local
ln -s ../lib local/lib

@bukzor
bukzor commented Apr 13, 2011

The culprit is this line at /usr/lib/python2.7/sysconfig.py:285

return os.path.join(get_path('stdlib').replace("/usr/local","/usr",1), "config" + (sys.pydebug and "_d" or ""), "Makefile")

There is no '/usr' in the stdlib path under virtualenv, so the string replace above doesn't do anything. I don't know whose bug this is really, or how best to fix it, so I'll leave it up to you all from here.

@bukzor bukzor closed this Apr 13, 2011
@bukzor bukzor reopened this Apr 13, 2011
@pshields

I encountered this issue as well. The previous poster's quick fix worked for me as a temporary solution.

@scottza
scottza commented Jun 1, 2011

It looks like the function path_locations() in virtualenv.py needs to be special cased for Debian based systems with Python 2.7.

On my Ubuntu 11.04 machine

import sysconfig
print(sysconfig.get_paths())
{'platinclude': '/usr/local/include/python2.7', 'platstdlib': '/usr/local/lib/python2.7', 'platlib': '/usr/local/lib/python2.7/dist-packages', 'purelib': '/usr/local/lib/python2.7/dist-packages', 'stdlib': '/usr/local/lib/python2.7', 'scripts': '/usr/local/bin', 'include': '/usr/local/include/python2.7', 'data': '/usr/local'}

virtualenv puts stuff in $VIRTUAL_ENV/lib, $VIRTUAL_ENV/include etc.

My version of bukzor's workaround is:

Activate the virtualenv
mkdir $VIRTUAL_ENV/local
ln -s $VIRTUAL_ENV/lib $VIRTUAL_ENV/local/lib
Install Numpy

@wgx731
wgx731 commented Jun 8, 2011

I want to install numpy with virtualenv on Ubuntu 11.04 as well. Encounter error. This is my log:

Downloading/unpacking numpy
Running setup.py egg_info for package numpy
Running from numpy source directory./bin/sh: svnversion: not found
non-existing path in 'numpy/distutils': 'site.cfg'
F2PY Version 2
blas_opt_info:
blas_mkl_info:
libraries mkl,vml,guide not found in /home/wgx731/.virtualenvs/python2.7/lib
libraries mkl,vml,guide not found in /usr/local/lib
libraries mkl,vml,guide not found in /usr/lib
NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  NOT AVAILABLE

atlas_blas_info:
  libraries f77blas,cblas,atlas not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/lib/sse2
  libraries f77blas,cblas,atlas not found in /usr/lib
  NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1417: UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
blas_info:
  libraries blas not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries blas not found in /usr/local/lib
  libraries blas not found in /usr/lib
  NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1426: UserWarning:
    Blas (http://www.netlib.org/blas/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [blas]) or by setting
    the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
blas_src_info:
  NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1429: UserWarning:
    Blas (http://www.netlib.org/blas/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [blas_src]) or by setting
    the BLAS_SRC environment variable.
  warnings.warn(BlasSrcNotFoundError.__doc__)
  NOT AVAILABLE

/bin/sh: svnversion: not found
lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /usr/lib
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries lapack_atlas not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2
  libraries lapack_atlas not found in /usr/lib/sse2
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_threads_info
  NOT AVAILABLE

atlas_info:
  libraries f77blas,cblas,atlas not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries lapack_atlas not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/lib/sse2
  libraries lapack_atlas not found in /usr/lib/sse2
  libraries f77blas,cblas,atlas not found in /usr/lib
  libraries lapack_atlas not found in /usr/lib
numpy.distutils.system_info.atlas_info
  NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1333: UserWarning:
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
lapack_info:
  libraries lapack not found in /home/wgx731/.virtualenvs/python2.7/lib
  libraries lapack not found in /usr/local/lib
  libraries lapack not found in /usr/lib
  NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1344: UserWarning:
    Lapack (http://www.netlib.org/lapack/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [lapack]) or by setting
    the LAPACK environment variable.
  warnings.warn(LapackNotFoundError.__doc__)
lapack_src_info:
  NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1347: UserWarning:
    Lapack (http://www.netlib.org/lapack/) sources not found.
    Directories to search for the sources can be specified in the
    numpy/distutils/site.cfg file (section [lapack_src]) or by setting
    the LAPACK_SRC environment variable.
  warnings.warn(LapackSrcNotFoundError.__doc__)
  NOT AVAILABLE

build_src
building py_modules sources
building library "npymath" sources
error: None
Complete output from command python setup.py egg_info:
Running from numpy source directory./bin/sh: svnversion: not found

non-existing path in 'numpy/distutils': 'site.cfg'

F2PY Version 2

blas_opt_info:

blas_mkl_info:

libraries mkl,vml,guide not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries mkl,vml,guide not found in /usr/local/lib

libraries mkl,vml,guide not found in /usr/lib

NOT AVAILABLE

atlas_blas_threads_info:

Setting PTATLAS=ATLAS

libraries ptf77blas,ptcblas,atlas not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib

libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2

libraries ptf77blas,ptcblas,atlas not found in /usr/lib

NOT AVAILABLE

atlas_blas_info:

libraries f77blas,cblas,atlas not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries f77blas,cblas,atlas not found in /usr/local/lib

libraries f77blas,cblas,atlas not found in /usr/lib/sse2

libraries f77blas,cblas,atlas not found in /usr/lib

NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1417: UserWarning:

Atlas (http://math-atlas.sourceforge.net/) libraries not found.

Directories to search for the libraries can be specified in the

numpy/distutils/site.cfg file (section [atlas]) or by setting

the ATLAS environment variable.

warnings.warn(AtlasNotFoundError.doc)

blas_info:

libraries blas not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries blas not found in /usr/local/lib

libraries blas not found in /usr/lib

NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1426: UserWarning:

Blas (http://www.netlib.org/blas/) libraries not found.

Directories to search for the libraries can be specified in the

numpy/distutils/site.cfg file (section [blas]) or by setting

the BLAS environment variable.

warnings.warn(BlasNotFoundError.doc)

blas_src_info:

NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1429: UserWarning:

Blas (http://www.netlib.org/blas/) sources not found.

Directories to search for the sources can be specified in the

numpy/distutils/site.cfg file (section [blas_src]) or by setting

the BLAS_SRC environment variable.

warnings.warn(BlasSrcNotFoundError.doc)

NOT AVAILABLE

/bin/sh: svnversion: not found

lapack_opt_info:

lapack_mkl_info:

mkl_info:

libraries mkl,vml,guide not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries mkl,vml,guide not found in /usr/local/lib

libraries mkl,vml,guide not found in /usr/lib

NOT AVAILABLE

NOT AVAILABLE

atlas_threads_info:

Setting PTATLAS=ATLAS

libraries ptf77blas,ptcblas,atlas not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries lapack_atlas not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib

libraries lapack_atlas not found in /usr/local/lib

libraries ptf77blas,ptcblas,atlas not found in /usr/lib/sse2

libraries lapack_atlas not found in /usr/lib/sse2

libraries ptf77blas,ptcblas,atlas not found in /usr/lib

libraries lapack_atlas not found in /usr/lib

numpy.distutils.system_info.atlas_threads_info

NOT AVAILABLE

atlas_info:

libraries f77blas,cblas,atlas not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries lapack_atlas not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries f77blas,cblas,atlas not found in /usr/local/lib

libraries lapack_atlas not found in /usr/local/lib

libraries f77blas,cblas,atlas not found in /usr/lib/sse2

libraries lapack_atlas not found in /usr/lib/sse2

libraries f77blas,cblas,atlas not found in /usr/lib

libraries lapack_atlas not found in /usr/lib

numpy.distutils.system_info.atlas_info

NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1333: UserWarning:

Atlas (http://math-atlas.sourceforge.net/) libraries not found.

Directories to search for the libraries can be specified in the

numpy/distutils/site.cfg file (section [atlas]) or by setting

the ATLAS environment variable.

warnings.warn(AtlasNotFoundError.doc)

lapack_info:

libraries lapack not found in /home/wgx731/.virtualenvs/python2.7/lib

libraries lapack not found in /usr/local/lib

libraries lapack not found in /usr/lib

NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1344: UserWarning:

Lapack (http://www.netlib.org/lapack/) libraries not found.

Directories to search for the libraries can be specified in the

numpy/distutils/site.cfg file (section [lapack]) or by setting

the LAPACK environment variable.

warnings.warn(LapackNotFoundError.doc)

lapack_src_info:

NOT AVAILABLE

/home/wgx731/.virtualenvs/python2.7/build/numpy/numpy/distutils/system_info.py:1347: UserWarning:

Lapack (http://www.netlib.org/lapack/) sources not found.

Directories to search for the sources can be specified in the

numpy/distutils/site.cfg file (section [lapack_src]) or by setting

the LAPACK_SRC environment variable.

warnings.warn(LapackSrcNotFoundError.doc)

NOT AVAILABLE

running egg_info

running build_src

build_src

building py_modules sources

building library "npymath" sources

error: None


Command python setup.py egg_info failed with error code 1

I have tried scottza's solution, but it doesn't work for me.
Any way to fix this? Thx

@pnasrat
Contributor
pnasrat commented Jun 8, 2011

Thanks, for the update. We're aware of this issue and I hope to look at in more depth this weekend.

@flaviovdf

Managed to get it working with the following workaround:

mkdir $VIRTUAL_ENV/local
ln -s $VIRTUAL_ENV/lib $VIRTUAL_ENV/local/lib
ln -s $VIRTUAL_ENV/include $VIRTUAL_ENV/local/include

@jezdez jezdez added a commit that closed this issue Jul 16, 2011
@jezdez jezdez Create an additional symlink to the 'local' dir in case the posix_loc…
…al install scheme is used. Fixes #118.
285679c
@jezdez jezdez closed this in 285679c Jul 16, 2011
@robertmuil

This fix is broken in certain setups.

If the python virtualenv path includes ...usr/local... then the virtualenv setup script will erroneously 'correct' it to ...local...

So, moving the virtualenv will avoid the problem: worth knowing for anyone who likes to install things under $HOME/usr.

@mikofski
mikofski commented Apr 2, 2012

still can't install numpy under virtualenv, but I also couldn't install it with pip just in system env.
virtualenv 1.6.4
ubuntu 11.10
pip 1.0
distribute 0.6.16

@dmckean
dmckean commented Aug 17, 2013

Something in sysconfig in python 2.7 seems to be at fault.

Python 2.7.3 (default, Apr 10 2013, 06:20:15)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_vars()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/sysconfig.py", line 476, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/lib/python2.7/sysconfig.py", line 344, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open /usr/django/app/my_env/lib/python2.7/config/Makefile (No such file or directory)
>>> sysconfig.get_config_vars()
{'py_version_nodot': '27', 'exec_prefix': '/usr/local/django/app/my_env', 'platbase': '/usr/local/django/app/my_env', 'py_version_short': '2.7', 'prefix': '/usr/local/django/app/my_env', 'base': '/usr/local/django/app/my_env', 'projectbase': '/usr/local/django/app/my_env/bin', 'py_version': '2.7.3'}
>>>
@nicksloan

This issue is caused by a patch applied by Ubuntu that specifically replaced /usr/local with /usr in several functions. Line 63404 in https://launchpadlibrarian.net/151545955/python2.7_2.7.3-5ubuntu4.3.diff.gz.

This stinks of taking a shortcut to support Ubuntu's deviations from the typical posix_local schema. Annoying.

For future reference: moving your virtualenv out of /usr/local... will fix it.

@brianz
brianz commented Nov 21, 2013

Hit this on Ubunu 12.04 and pip 1.4.1, virtualenv 1.10.1....moving the virtualenv out of /usr/local did the trick.

@Neetuj
Neetuj commented Apr 25, 2014

running into the same issue
root@njain2:/usr/local/horizon# pip --version
pip 1.5.4 from /usr/local/lib/python2.7/dist-packages (python 2.7)
root@njain2:/usr/local/horizon# python --version
Python 2.7.3
root@njain2:/usr/local/horizon# virtualenv --version
1.7.1.2

File "/usr/lib/python2.7/sysconfig.py", line 344, in _init_posix

raise IOError(msg)
IOError: invalid Python installation: unable to open /usr/horizon/.venv/lib/python2.7/config/Makefile (No such file or directory)

Cleaning up...
Command python setup.py egg_info failed with error code 1 in /usr/local/horizon/.venv/build/cryptography
Storing debug log for failure in /root/.pip/pip.log

@Neetuj
Neetuj commented Apr 25, 2014

the file exists in /usr/local ..but i am not sure how to make it read from there .. and where exaclty it should be

root@njain2:/usr/local/horizon# ls /usr/horizon/.venv/lib/python2.7/config/Makefile
ls: cannot access /usr/horizon/.venv/lib/python2.7/config/Makefile: No such file or directory
root@njain2:/usr/local/horizon# ls /usr/local/horizon/.venv/lib/python2.7/config/Makefile
/usr/local/horizon/.venv/lib/python2.7/config/Makefile

@pkpp1233

i'm running into this same issue trying to install numpy in a docker container.

@bukzor
bukzor commented Nov 14, 2014

@Neetuj: The culprit for you is still this line at /usr/lib/python2.7/sysconfig.py:285, shown above, although for a different reason.

@bukzor
bukzor commented Nov 14, 2014

This was apparently an ill-conceived patch released by ubuntu for precise:
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/precise/python2.7/precise-security/view/head:/debian/patches/distutils-install-layout.diff#L282

The replace statement is gone in trusty, but the scheme is still set to 'posix-local', so virtualenv's workaround persists, causing bugs in the coveragepy test suite under ubuntu:

https://bitbucket.org/ned/coveragepy/issue/337/test-failures-on-tip#comment-13653960

@naarkhoo

I have the same problem - for installing Pilow on virtualenv

creating pip-egg-info/Pillow.egg-info

writing pip-egg-info/Pillow.egg-info/PKG-INFO

writing top-level names to pip-egg-info/Pillow.egg-info/top_level.txt

writing dependency_links to pip-egg-info/Pillow.egg-info/dependency_links.txt

writing manifest file 'pip-egg-info/Pillow.egg-info/SOURCES.txt'

warning: manifest_maker: standard file '-c' not found

error: invalid Python installation: unable to open /usr/lib/python2.7/config/Makefile (No such file or directory)


Cleaning up...
Command python setup.py egg_info failed with error code 1 in /home/rostam/.virtualenvs/myPelican/build/Pillow
Storing debug log for failure in /home/rostam/.pip/pip.log

Is there any workaround ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment