How do you build kaldi/lib/_clif.so?
#72
Comments
|
The cmake incantation for building How are you building pykaldi? Aren't you using cmake like we do? It seems like you are building the clif shared library and linking against it but the python modules cannot find it at run time, since you are getting an import error instead of a linker error. We set the RPATH of pykaldi modules here so that they can find other pykaldi modules and the clif shared library inside the installed package. |
|
I am building with cmake, but the difference is that I want a wheel runnable on a different machine. So I run Next, I'm using the auditwheel tool to include all .so files inside the wheel. I was able to embed |
|
I am guessing that is because |
|
|
|
I am guessing this problem has to do with auditwheel editing RPATH of extension modules. Checking the RPATH of shared libs after they are modified by auditwheel might give a clue about what is going wrong. I tried to run Can you try running the following on your end? This is what I get when I run it on the vanilla wheel. |
|
For the modified wheel it is: |
|
Yeah it seems auditwheel is not appending to RPATH but instead overwriting it. Relevant issue. |
|
Can you run the following as well? |
|
I think we can fix the RPATH problem by editing this function. I think all needs to be done is to set RPATH to instead of |
|
Thanks! This is very helpful! |
|
|
I think we can get away with the following monkey patch until auditwheel fixes their code. This preserves RPATH directories that start with cat ./auditwheel
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re
import sys
import logging
from os.path import relpath, dirname
from os.path import join as pjoin
from subprocess import check_call, check_output
from auditwheel.main import main
from auditwheel import repair
logger = logging.getLogger('auditwheel.repair')
def patchelf_set_rpath(fn, libdir):
rpath = check_output(['patchelf', '--print-rpath', fn]).decode("utf-8").strip()
rpath = list(filter(lambda x: x.startswith('$ORIGIN'), rpath.split(":")))
rpath.append(pjoin('$ORIGIN', relpath(libdir, dirname(fn))))
rpath = ":".join(rpath)
logger.debug('Setting RPATH: %s to "%s"', fn, rpath)
check_call(['patchelf', '--force-rpath', '--set-rpath', rpath, fn])
repair.patchelf_set_rpath = patchelf_set_rpath
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(main()) |
|
Yay! It worked! |
|
Are you interested in a script building wheel? I could submit a PR |
|
Very much interested! Are you building it with the manylinux docker image? |
|
Not yet. So far I tried only with pykaldi/pykaldi docker and |
|
Got it. Maybe we can build the wheel and upload it somewhere as part of docker image build on Travis. |
|
@dmitriy-serdyuk I just pushed a change which might affect redistributable wheels. After this change, |
|
Great, thanks! |
|
Unfortunately python library is still needed to compile CLIF. manylinux docker is a huge pain to work with, they don't have Here is dockerfile (not working) so far: |
|
Thanks for the effort you put in! I don't know if CLIF people had a reason to link with libpython. We are building and packaging the same CLIF runtime library (which is the part linking with libpython) as part of pykaldi and we don't need to link against libpython. To be honest, current state of CLIF is the reason why I haven't invested more time into distributing pykaldi wheels. I was hoping they would release a new version of CLIF that you could install from a package manager and we wouldn't have to deal with these issues when distributing pykaldi. I was also hoping that would happen some time last year but nothing yet. Do you need to build on Centos 5? Who is still using that? :) @vrmpx is building our conda package on Centos 7 and that seems to work for most people. manylinux2010 image is Centos 6 which might be significantly easier to work with. I think even pytorch people are building their packages on Centos 6. |
I am trying to build a wheel for this package. I managed to encapsulate kaldi shared lib into the wheel using auditwheel. Unfortunately it still fails with
I don't understand how you build
kaldi/lib/_clif.sofrom sources inpykaldi/kaldi/lib/clif/python/. Is there some code missing in the repo?The text was updated successfully, but these errors were encountered: