New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip install in virtualenv installs lib to wrong directory #445

Open
praseodym opened this Issue Aug 10, 2015 · 11 comments

Comments

Projects
None yet
9 participants
@praseodym

praseodym commented Aug 10, 2015

I did a pip install capstone in my virtualenv, and the lib was installed in the wrong directory, which causes the following error:

>>> import capstone
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxxxx/.virtualenvs/yyyyy/local/lib/python2.7/site-packages/capstone/__init__.py", line 230, in <module>
    raise ImportError("ERROR: fail to load the dynamic library.")
ImportError: ERROR: fail to load the dynamic library.

Python files were installed in in /home/xxxxx/.virtualenvs/yyyyy/lib/python2.7/site-packages/capstone/*.py(c)
Library file was installed in /home/xxxxx/.virtualenvs/yyyyy/lib/python2.7/site-packages/home/xxxxx/.virtualenvs/yyyyy/lib/python2.7/site-packages/capstone/libcapstone.so

Moving libcapstone.so to the same directory as the Python files fixed the problem.

@aquynh

This comment has been minimized.

Show comment
Hide comment
@aquynh

aquynh Aug 10, 2015

Owner

@achernet, can you please look at this issue?

i guess this happens with v3.0.4, related to your PR #406

thanks.

Owner

aquynh commented Aug 10, 2015

@achernet, can you please look at this issue?

i guess this happens with v3.0.4, related to your PR #406

thanks.

@vvvverre

This comment has been minimized.

Show comment
Hide comment
@vvvverre

vvvverre Sep 17, 2015

Just chiming in to say that I just experienced the same issue. Moving the .so file as mentioned by @praseodym worked for me as well.

If it is any help in debugging, my setup is Arch Linux, with virtualenv 13.1.2, pip 7.1.2 & python 2.7.10 or python 3.4.3. I was able to reproduce this in newly created virtualenvs.

vvvverre commented Sep 17, 2015

Just chiming in to say that I just experienced the same issue. Moving the .so file as mentioned by @praseodym worked for me as well.

If it is any help in debugging, my setup is Arch Linux, with virtualenv 13.1.2, pip 7.1.2 & python 2.7.10 or python 3.4.3. I was able to reproduce this in newly created virtualenvs.

@0x64616E69656C

This comment has been minimized.

Show comment
Hide comment
@0x64616E69656C

0x64616E69656C Jan 8, 2016

I have the same issue on cygwin when using pip install capstone.
The capstone.dll is installed to /usr/lib/python2.7/site-packages/usr/lib/python2.7/site-packages/capstone, so there's a / missing in the installation path. copying the file to /usr/lib/python2.7/site-packages/capstone fixes that problem.

//edit: works fine on debian though.

0x64616E69656C commented Jan 8, 2016

I have the same issue on cygwin when using pip install capstone.
The capstone.dll is installed to /usr/lib/python2.7/site-packages/usr/lib/python2.7/site-packages/capstone, so there's a / missing in the installation path. copying the file to /usr/lib/python2.7/site-packages/capstone fixes that problem.

//edit: works fine on debian though.

@aquynh

This comment has been minimized.

Show comment
Hide comment
@aquynh

aquynh Jan 11, 2016

Owner

sorry that i dont use Arch Linux. if anyone knows how to fix this, please send a pull request, thanks.

Owner

aquynh commented Jan 11, 2016

sorry that i dont use Arch Linux. if anyone knows how to fix this, please send a pull request, thanks.

@pidydx

This comment has been minimized.

Show comment
Hide comment
@pidydx

pidydx Jan 29, 2016

This is happening on ubuntu with capstone 3.0.4 as well. The package installs to /usr/local/lib/python2.7/dist-packages/capstone, but then it also puts libcapstone.so in /usr/local/lib/python2.7/dist-packages/usr/lib/python2.7/dist-packages/capstone/libcapstone.so

Using pip install on 3.0.3 everything works fine with the python module installed to /usr/local/lib/python2.7/dist-packages/capstone and the libcapstone.so installed to /usr/lib/python2.7/dist-packages/capstone/libcapstone.so

Edit:
Removing 3.0.4 and reinstalling it with pip appears to have put everything in the right place. It seems like something may be happening with an environment variable under certain conditions copying files to the wrong place.

pidydx commented Jan 29, 2016

This is happening on ubuntu with capstone 3.0.4 as well. The package installs to /usr/local/lib/python2.7/dist-packages/capstone, but then it also puts libcapstone.so in /usr/local/lib/python2.7/dist-packages/usr/lib/python2.7/dist-packages/capstone/libcapstone.so

Using pip install on 3.0.3 everything works fine with the python module installed to /usr/local/lib/python2.7/dist-packages/capstone and the libcapstone.so installed to /usr/lib/python2.7/dist-packages/capstone/libcapstone.so

Edit:
Removing 3.0.4 and reinstalling it with pip appears to have put everything in the right place. It seems like something may be happening with an environment variable under certain conditions copying files to the wrong place.

@aquynh

This comment has been minimized.

Show comment
Hide comment
@aquynh

aquynh Mar 29, 2016

Owner

the latest Github code supposed to fix this issue. can you guys please confirm?

thanks

Owner

aquynh commented Mar 29, 2016

the latest Github code supposed to fix this issue. can you guys please confirm?

thanks

@Sjlver

This comment has been minimized.

Show comment
Hide comment
@Sjlver

Sjlver Mar 29, 2016

What would be the command to test this? I tried the following in a fresh virtualenv:

pip install 'git+git://github.com/aquynh/capstone.git#egg=capstone&subdirectory=bindings/python'

Pip says that it successfully installed capstone-3.0.4, but it doesn't actually compile any library.

Sjlver commented Mar 29, 2016

What would be the command to test this? I tried the following in a fresh virtualenv:

pip install 'git+git://github.com/aquynh/capstone.git#egg=capstone&subdirectory=bindings/python'

Pip says that it successfully installed capstone-3.0.4, but it doesn't actually compile any library.

@raimue

This comment has been minimized.

Show comment
Hide comment
@raimue

raimue Mar 30, 2016

I noticed that ./setup.py sdist does not actually include the sources into the tarball, although they were hardlinked to the src/ directory:

$ ./setup.py sdist
...
$ tar tf dist/capstone-3.0.4.tar.gz |grep src
$

That could be solved by adding recursive-include src * to MANIFEST.in.

However, this is not the full solution. When running commands such as ./setup.py bdist or ./setup.py bdist_wheel (with src/ in place), they will actually build the library, but the library file itself will not be included in the generated package. I don't understand python packaging enough to debug this, but apparently the package_data specification does not work as expected.

raimue commented Mar 30, 2016

I noticed that ./setup.py sdist does not actually include the sources into the tarball, although they were hardlinked to the src/ directory:

$ ./setup.py sdist
...
$ tar tf dist/capstone-3.0.4.tar.gz |grep src
$

That could be solved by adding recursive-include src * to MANIFEST.in.

However, this is not the full solution. When running commands such as ./setup.py bdist or ./setup.py bdist_wheel (with src/ in place), they will actually build the library, but the library file itself will not be included in the generated package. I don't understand python packaging enough to debug this, but apparently the package_data specification does not work as expected.

@aquynh

This comment has been minimized.

Show comment
Hide comment
@aquynh

aquynh Mar 31, 2016

Owner

@scudette, do you have any comments?

Owner

aquynh commented Mar 31, 2016

@scudette, do you have any comments?

@scudette

This comment has been minimized.

Show comment
Hide comment
@scudette

scudette Mar 31, 2016

Contributor

Hi raimue,
You are right, the src needs to be added to the MANIFEST.in - I sent a PR to the next branch with it but I did not send it to the master branch. The way this works is that you always build a source distribution and then compile and install this source distribution.

Also make sure you properly remove the build and dist directories because setuptools tries to cache things and it it finds something there is will not rebuild the .so. Its easier to make it rebuild both sdist and bdist at the same time:

python setup.py sdist bdist_wheel

I think there is still a problem though. If I try to build a wheel with pip:

pip wheel .

It will actually build the .so but it will not include it because it seems that package_data is expanded before build_clib is run and it really does not know about the .so file at all. If I add a dummy .so file to the capstone directory this does not work as it copies the dummy file before it builds it.

I have recently done some work on understanding the semantics of the data_files directive and it may be that this is the better solution. The main problem with it is that you can not predict exactly where the .so file will end up getting installed to so you must use pkg_resources to find it. In turn pkg_resources is kind of complicated to use because it only half works for all the cases it is supposed to work on (e.g. it doesnt work right for pyinstaller and dynamic imports):

https://github.com/google/rekall/blob/master/rekall-core/rekall/resources.py

Finally I think the solution I came up with for capstone here:
https://github.com/scudette/rekall-capstone

May be better because it sidesteps the entire issue - the .so is build in the same way as a python extension module. This has to work for all packaging systems because its the same as copying a normal c extension module into the package. The main difference is that it does not use cmake at all and just builds it by itself.

Maybe it is better to extend build_ext instead of extending build_clib in capstone?

Contributor

scudette commented Mar 31, 2016

Hi raimue,
You are right, the src needs to be added to the MANIFEST.in - I sent a PR to the next branch with it but I did not send it to the master branch. The way this works is that you always build a source distribution and then compile and install this source distribution.

Also make sure you properly remove the build and dist directories because setuptools tries to cache things and it it finds something there is will not rebuild the .so. Its easier to make it rebuild both sdist and bdist at the same time:

python setup.py sdist bdist_wheel

I think there is still a problem though. If I try to build a wheel with pip:

pip wheel .

It will actually build the .so but it will not include it because it seems that package_data is expanded before build_clib is run and it really does not know about the .so file at all. If I add a dummy .so file to the capstone directory this does not work as it copies the dummy file before it builds it.

I have recently done some work on understanding the semantics of the data_files directive and it may be that this is the better solution. The main problem with it is that you can not predict exactly where the .so file will end up getting installed to so you must use pkg_resources to find it. In turn pkg_resources is kind of complicated to use because it only half works for all the cases it is supposed to work on (e.g. it doesnt work right for pyinstaller and dynamic imports):

https://github.com/google/rekall/blob/master/rekall-core/rekall/resources.py

Finally I think the solution I came up with for capstone here:
https://github.com/scudette/rekall-capstone

May be better because it sidesteps the entire issue - the .so is build in the same way as a python extension module. This has to work for all packaging systems because its the same as copying a normal c extension module into the package. The main difference is that it does not use cmake at all and just builds it by itself.

Maybe it is better to extend build_ext instead of extending build_clib in capstone?

mossberg added a commit to trailofbits/manticore that referenced this issue Feb 13, 2017

mossberg added a commit to trailofbits/manticore that referenced this issue Feb 13, 2017

Remove embedded elftools, use proper dependency (#1)
* Import elftools properly

* Add pyelftools as dep

* Use elftools dep

* Rm embedded elftools

* Make travis use requirements.txt

* Workaround capstone install bug

aquynh/capstone#445
@rupran

This comment has been minimized.

Show comment
Hide comment
@rupran

rupran Jul 31, 2018

This appears to be finally fixed in 3.0.5 by commit ced66e3, right?

At least for me, installing capstone inside a virtualenv and doing import capstone works out of the box now, as the library is placed in a lib/ subdirectory at the correct location.

rupran commented Jul 31, 2018

This appears to be finally fixed in 3.0.5 by commit ced66e3, right?

At least for me, installing capstone inside a virtualenv and doing import capstone works out of the box now, as the library is placed in a lib/ subdirectory at the correct location.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment