pip install in virtualenv installs lib to wrong directory

[praseodym]

I did a pip install capstone in my virtualenv, and the lib was installed in the wrong directory, which causes the following error:

>>> import capstone
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxxxx/.virtualenvs/yyyyy/local/lib/python2.7/site-packages/capstone/__init__.py", line 230, in <module>
    raise ImportError("ERROR: fail to load the dynamic library.")
ImportError: ERROR: fail to load the dynamic library.

Python files were installed in in /home/xxxxx/.virtualenvs/yyyyy/lib/python2.7/site-packages/capstone/*.py(c)
Library file was installed in /home/xxxxx/.virtualenvs/yyyyy/lib/python2.7/site-packages/home/xxxxx/.virtualenvs/yyyyy/lib/python2.7/site-packages/capstone/libcapstone.so

Moving libcapstone.so to the same directory as the Python files fixed the problem.

[aquynh]

@achernet, can you please look at this issue?

i guess this happens with v3.0.4, related to your PR #406

thanks.

[vvvverre]

Just chiming in to say that I just experienced the same issue. Moving the .so file as mentioned by @praseodym worked for me as well.

If it is any help in debugging, my setup is Arch Linux, with virtualenv 13.1.2, pip 7.1.2 & python 2.7.10 or python 3.4.3. I was able to reproduce this in newly created virtualenvs.

[ghost]

I have the same issue on cygwin when using pip install capstone.
The capstone.dll is installed to /usr/lib/python2.7/site-packages/usr/lib/python2.7/site-packages/capstone, so there's a / missing in the installation path. copying the file to /usr/lib/python2.7/site-packages/capstone fixes that problem.

//edit: works fine on debian though.

[aquynh]

sorry that i dont use Arch Linux. if anyone knows how to fix this, please send a pull request, thanks.

[pidydx]

This is happening on ubuntu with capstone 3.0.4 as well. The package installs to /usr/local/lib/python2.7/dist-packages/capstone, but then it also puts libcapstone.so in /usr/local/lib/python2.7/dist-packages/usr/lib/python2.7/dist-packages/capstone/libcapstone.so

Using pip install on 3.0.3 everything works fine with the python module installed to /usr/local/lib/python2.7/dist-packages/capstone and the libcapstone.so installed to /usr/lib/python2.7/dist-packages/capstone/libcapstone.so

Edit:
Removing 3.0.4 and reinstalling it with pip appears to have put everything in the right place. It seems like something may be happening with an environment variable under certain conditions copying files to the wrong place.

[aquynh]

the latest Github code supposed to fix this issue. can you guys please confirm?

thanks

[Sjlver]

What would be the command to test this? I tried the following in a fresh virtualenv:

pip install 'git+git://github.com/aquynh/capstone.git#egg=capstone&subdirectory=bindings/python'

Pip says that it successfully installed capstone-3.0.4, but it doesn't actually compile any library.

[raimue]

I noticed that ./setup.py sdist does not actually include the sources into the tarball, although they were hardlinked to the src/ directory:

$ ./setup.py sdist
...
$ tar tf dist/capstone-3.0.4.tar.gz |grep src
$

That could be solved by adding recursive-include src * to MANIFEST.in.

However, this is not the full solution. When running commands such as ./setup.py bdist or ./setup.py bdist_wheel (with src/ in place), they will actually build the library, but the library file itself will not be included in the generated package. I don't understand python packaging enough to debug this, but apparently the package_data specification does not work as expected.

[aquynh]

@scudette, do you have any comments?

[scudette]

Hi raimue,
You are right, the src needs to be added to the MANIFEST.in - I sent a PR to the next branch with it but I did not send it to the master branch. The way this works is that you always build a source distribution and then compile and install this source distribution.

Also make sure you properly remove the build and dist directories because setuptools tries to cache things and it it finds something there is will not rebuild the .so. Its easier to make it rebuild both sdist and bdist at the same time:

python setup.py sdist bdist_wheel

I think there is still a problem though. If I try to build a wheel with pip:

pip wheel .

It will actually build the .so but it will not include it because it seems that package_data is expanded before build_clib is run and it really does not know about the .so file at all. If I add a dummy .so file to the capstone directory this does not work as it copies the dummy file before it builds it.

I have recently done some work on understanding the semantics of the data_files directive and it may be that this is the better solution. The main problem with it is that you can not predict exactly where the .so file will end up getting installed to so you must use pkg_resources to find it. In turn pkg_resources is kind of complicated to use because it only half works for all the cases it is supposed to work on (e.g. it doesnt work right for pyinstaller and dynamic imports):

https://github.com/google/rekall/blob/master/rekall-core/rekall/resources.py

Finally I think the solution I came up with for capstone here:
https://github.com/scudette/rekall-capstone

May be better because it sidesteps the entire issue - the .so is build in the same way as a python extension module. This has to work for all packaging systems because its the same as copying a normal c extension module into the package. The main difference is that it does not use cmake at all and just builds it by itself.

Maybe it is better to extend build_ext instead of extending build_clib in capstone?

[rupran]

This appears to be finally fixed in 3.0.5 by commit ced66e3, right?

At least for me, installing capstone inside a virtualenv and doing import capstone works out of the box now, as the library is placed in a lib/ subdirectory at the correct location.