Unable to install compiled Python modules under pyenv on OS X 10.10 #273
Comments
|
I'm using OS X 10.10.1 and confirmed I can install % sw_vers
ProductName: Mac OS X
ProductVersion: 10.10.1
BuildVersion: 14B25The actual error is as follows. I don't know why it's trying to use |
|
pyenv will find brew's readline and try to use it if installed. Basically, you don't have to do I couldn't confirm this build error of |
|
Thanks for looking into this and for the info about When I generated the error message I was using a box with clean install of 10.10.1 (erased hard drive before install), and I did not set any special options when building CPython. On my primary computer, I upgraded to 10.10.1 without erasing hard drive and without a new install of brew, pyenv, etc. and everything works fine. However, when I try to install this stuff on a clean-install of 10.10.1, I get the errors noted above. |
|
I confirmed same happens on my OS X 10.10.1, at least, with CPython |
|
I can confirm I run into the same issue (and error message) with gevent on python 2.7.5 Digging a bit more into running the build manually step by step, I did get this specific step to not error-out by:
But I don't know what needs to be done to get the build process to use that instead of trying python.exe |
|
I fixed this by doing something odd: Now the line in question looks like: |
|
I repeated the installation described in my initial comment, this time using Python 2.7.8 rather than 2.7.6 (and also skipping the |
|
This is definitely not a pyenv specific issue. I've confirmed I get the same build issues with python < 2.7.8 outside of pyenv. So I think this can be closed. |
|
@conslo Agreed. This isn't a pyenv specific issue. Thanks for you investigation 😃 |
|
I met with the same problem, and when I switched from 2.7.5 (installed by pyenv) to system it works. I guess it's still related to pyenv. I notice that when using the system version, the 'clang' command doesn't have an argument UPDATE: switch to system I means, |
|
I got the same error when compiling python libraries while using python < 2.7.8 without pyenv installed at all. @shivawu which package did you try installing while under the system 2.7.5? |
|
I was trying to install pyyaml, 3.10, and my system is 10.10.1, python On Fri, Dec 19, 2014 at 2:51 PM, Travis Johnson notifications@github.com
Chenyang WU |
|
I'm on 10.10.1 as well, I'll experiment with this tonight and see if I can run down a cause |
|
Interesting. I found % PYENV_VERSION=2.7.7 python -c 'import sysconfig;print(sysconfig.get_config_var("BLDSHARED"))'
clang -bundle -bundle_loader python.exe -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/yyuu/.pyenv/versions/2.7.7/libIn CPython 2.7.8, unlike 2.7.7, it doesn't have % PYENV_VERSION=2.7.8 python -c 'import sysconfig;print(sysconfig.get_config_var("`"))'
clang -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/yyuu/.pyenv/versions/2.7.8/libI need more investigation on usage of |
|
I wonder if this was a bad comparison of |
|
This is exactly what I am observing. Even if we can't push for a backport, I wonder if there's a possibility of a workaround? Like make python think it's 10.9 when compiling things? |
|
I was able to work around this issue by building pip wheels for all my requirements (specifically PyCrypto) in 2.7.8, and installing from wheels in a 2.7.6 virtual environment. |
|
@kennytrytek-wf 's solution also worked for me. Here is what I needed to do to get this working: # This fails, with the python.exe error
pyenv global 2.7.6
pip install -r requirements.txt
# This works
pyenv global 2.7.9
pip wheel -r requirements.txt
pyenv global 2.7.6
pip install -r requirements.txt |
|
@conslo I had exact the same issue. |
|
Hi, I hit this problem while trying to setup my Ansible dev environment locally: Did anyone find a solution to this? None of the suggestions made on this thread worked for me. I am able to install Any help will be appreciated. Thanks in advance. Angel |
|
@hebbo The issue is actually a flaw in the way python detects what operating system you're using (having multiple digits after the version dot seems to break it, ie: 10.10, 10.11). And, as it is with bugs, the newer versions of python have it fixed. This is why 2.7.8 works but not 2.7.6 etc. The older versions of python will not work because they are broken. |
|
@conslo Seems we will have to switch to 2.7.8 then if we waant to manage our local envs using |
|
Perhaps the fix could be backported into pyenv's python builds <2.7.8 since it breaks functionality, rendering pyenv useless for projects using those versions (unfortunately, we don't always have control over such constraints). |
|
@mikenerone I don't think "backporting to an existing patch release" makes sense. Backporting to an existing feature release, sure (which would create a new patch release). But 2.7 is the only Python 2 version that's even supported right now. Versions are meant to be immutable. If you change something and release the change then that's different software than the previous release, so it's a new version. You wouldn't want to automatically replace the 2.7.6 version that people are using out from under them by releasing new software with the same version number anymore than you'd want to implicitly upgrade people to new versions (which, as you describe, is something some places avoid)... because that's the same thing. The process of upgrading from the 2.7.x you're on to, say, 2.7.11 (the current release) should be the same process you'd have to go through to upgrade to some 2.7.x.1 release sub-release-thing that you're requesting. |
|
@conslo I'm normally in total agreement with your sentiment here, but in this case the result is that one's code can be perfectly fine, and yet locally with pyenv it doesn't work at all, which is already quite a difference in behavior. If one's target environment is, say, 2.7.5 and can't be changed immediately just to accommodate pyenv, the only local workaround right now is to use 2.7.8 for development instead and hope that's "close enough", which is what people are doing. Just patching this one backport into earlier versions (and pyenv does already have the precedent of patching the Python sources, btw) is much less likely to expose a substantive behavioral difference that that. Sadly, there are no good answers here, including the "do nothing" answer. IMO, the least harmful answer should therefore be selected, and "it doesn't work at all" is the most harmful. |
|
@mikenerone honestly I had no idea pyenv had a precedent of patching sources. If that's the case it seems viable. Very early in the thread I even suggested pyenv could perhaps do something to make python think it was on an earlier version of OSX during the compile. Then perhaps someone should find the relevant patch and link it :) (as an aside though, you should really look into updating the version of python you're using, 2.7.5 is 3 1/2 years older than the current 2.7.11, and they're point releases, so unlikely to break much if anything (though I'm assuming you have tests to check). On that note though 2.7.11 made some internal changes to ssl implementations, which in particular broke older versions of greenlet and thus gevent, so I'd suggest 2.7.10 as that's even less likely to break stuff) |
|
@conslo I hear ya, but the root of the holdup for a lot of people is the policy of their particular distro (particularly the major ones). There are ways to achieve newer Python versions, of course, but many people feel like when they do that, they're throwing away the benefit of using a major, supported distro in the first place (that said, in answer to your aside, we bit the bullet and are in the process of moving to Py3 anyway). The fact that so many development targets are these major distros with older Pythons is the reason pyenv not being able to support those versions is a significant problem. Side note: those SSL changes you refer to were done in 2.7.9, and are already backported by those major distros, so that's usually not a concern (e.g. RedHat/CentOS have already done so with their current blessed Python, which is 2.7.5), so it's at least arguable that A. this sortof leaves pyenv as the only "real" hurdle for this situation, and B. it's hard to get away from those behavioral differences you mentioned regardless of what we do. ;) |
|
Interestingly enough, this problem seems to have disappeared for me. The only thing I can think of that I changed is that I switched from homebrew to MacPorts. I don't care enough to test it myself :), but if you're still affected, you might give that a try. |
|
When attempting to install FailureSuccessedit: We use such an old version of python for several reasons, namely due to support for Autodesk Maya among other 3D software packages. |
|
Well, Working solutionFirst, we confirm the issue $ pyenv install 2.7.6
$ pyenv virtualenv 2.7.6 testme
$ pyenv activate testme
$ pip install --no-cache lxml
...
ld: file not found: python.exe
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang' failed with exit status 1Then, as @yyuu has mentioned in this thread in 2014, that $ grep bundle_loader /Users/andrei/.pyenv/versions/2.7.6/lib/python2.7/_sysconfigdata.py
./lib/python2.7/_sysconfigdata.py: 'BLDSHARED': 'clang -bundle -bundle_loader python.exe -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl/lib -L/Users/andrei/.pyenv/versions/2.7.6/lib',
./lib/python2.7/_sysconfigdata.py: 'LDCXXSHARED': 'c++ -bundle -bundle_loader /Users/andrei/.pyenv/versions/2.7.6/bin/python2.7',
./lib/python2.7/_sysconfigdata.py: 'LDSHARED': 'clang -bundle -bundle_loader python.exe -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/openssl/lib -L/Users/andrei/.pyenv/versions/2.7.6/lib',Now, if we try removing $ sed -i -e "s/-bundle_loader python.exe//g" \
/Users/andrei/.pyenv/versions/2.7.6/lib/python2.7/_sysconfigdata.py
$ pip install --no-cache lxml
...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang' failed with exit status 1It seems needed, so we use another way. As you may have noticed, $ export p=/Users/andrei/.pyenv/versions/2.7.6
$ sed -i -e "s~-bundle_loader python.exe~-bundle_loader ${p}/bin/python2.7~g" \
"$p/lib/python2.7/_sysconfigdata.py"Or more aggressively, if you want to fix all $ sed -i -e "s~python.exe~${p}/bin/python2.7~g" "$p/lib/python2.7/_sysconfigdata.py"Now try installing: $ pip install --no-cache lxml
Collecting lxml
Downloading lxml-3.7.3.tar.gz (3.8MB)
100% |████████████████████████████████| 3.8MB 49.0MB/s
Installing collected packages: lxml
Running setup.py install for lxml ... done
Successfully installed lxml-3.7.3Done! |
|
@yyuu Maybe include such patch in the installer scripts? |
|
@yyuu That's similar to what Homebrew does https://github.com/Homebrew/homebrew-core/blob/master/Formula/python.rb#L206-L210 # Prevent third-party packages from building against fragile Cellar paths
inreplace [lib_cellar/"_sysconfigdata.py",
lib_cellar/"config/Makefile",
frameworks/"Python.framework/Versions/Current/lib/pkgconfig/python-2.7.pc"],
prefix, opt_prefix |
|
andreif's method worked for me in installing biopython via pip for Python 2.7.5 |
|
@andreif 's also worked for me for issues with pyyaml for pyenv + virtualenv with python 2.7.6, MacOS Sierra 10.12.5 |
|
@andreif's method also worked for me in trying to install pycrypto (as a needed requirement for ansible) with Python 2.7.5 on OSX El Capitan 10.11.c |
|
On macOS Sierra 10.12.6, this worked for me:
|
|
Spent 3 days searching everywhere for a solution, couldn't install psycopg2 lxml and paramiko with a specific version, and thanks to Andreif's solution, now it's working ! Thank you a lot !
|
Address #273 via a patch from python#21811
* 'master' of github.com:pyenv/pyenv: CPython 3.7.0b4 v1.2.4 Add pypy-portable 6.0.0 CPython 2.7.15 Add pypy 6.0.0 Refactor test code of python-build. Use curl during tests by default Import latest changes from https://github.com/rbenv/ruby-build as of v20180424 Address pyenv#273 via a patch from python#21811 Rewrite python-build tests with using `PYTHON_BUILD_HTTP_CLIENT` Allow overriding HTTP client type based on environment variable `PYTHON_BUILD_HTTP_CLIENT` (pyenv#1126) Add basic test for rehash wait Fix rehash test to give up sooner after lock file's presence Renamed variable; s/PYENV_REHASH_LOCK_TIMEOUT/PYENV_REHASH_TIMEOUT/ Experimental implementation to wait rehash until acquiring lock
thank you ,it works! |
I'm running into problems installing Python modules under pyenv on OS X 10.10.
Here's the scenario:
Start with a fresh install of OS X 10.10.
Get all software updates via App Store (OS X 10.10.1)
Install Xcode, run it once, and install its command line tools
xcode-select --install
Install homebrew and pyenv
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew doctor
brew install readline
brew link readline --force # Should I do this?
brew install pyenv
Install a Python and try to install a module. Many modules work fine (Python only), but those requiring compilation seem to fail (I used paramiko as the example)
pyenv install 2.7.6
pip install paramiko
Here is the failure message:
I realize this could be a pip issue; however, I'm able to install Python and modules like paramiko if I install the Python directly under homebrew, rather than via pyenv.
Any ideas? Thanks.
The rest of this is the pip log file:
The text was updated successfully, but these errors were encountered: