Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to install kaldifeat from source #59

Closed
AmirHussein96 opened this issue Nov 29, 2022 · 9 comments
Closed

Failed to install kaldifeat from source #59

AmirHussein96 opened this issue Nov 29, 2022 · 9 comments

Comments

@AmirHussein96
Copy link

I tried installing kaldifeat from source since both pip and conda installation failed.
I followed the suggestions in FAQ

export KALDIFEAT_CMAKE_ARGS="-DCMAKE_BUILD_TYPE=Release -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include"

Then I run the python setup.py install

I could not capture the complete logs but I attached the CMakeOutput.log, and CMakeError.log

The installation started well but then failed for some reason, I managed to capture some of the logs manually because
The command python setup.py install &>log gave file with unreadable data.

opying kaldifeat/python/kaldifeat/mfcc.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
copying kaldifeat/python/kaldifeat/fbank.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
copying kaldifeat/python/kaldifeat/offline_feature.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
copying kaldifeat/python/kaldifeat/__init__.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
copying kaldifeat/python/kaldifeat/online_feature.py -> build/lib.linux-x86_64-cpython-38/kaldifeat
running build_ext
Setting PYTHON_EXECUTABLE to /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python
For fast compilation, run:
export KALDIFEAT_MAKE_ARGS="-j"; python setup.py install
Setting make_args to '-j4'
build command is:

                cd build/temp.linux-x86_64-cpython-38

                cmake -DCMAKE_BUILD_TYPE=Release -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include -DPYTHON_EXECUTABLE=/home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python -Dkaldifeat_BUILD_TESTS=OFF  -DCMAKE_INSTALL_PREFIX=/alt-arabic/speech/amir/k2/tmp/kaldifeat/build/lib.linux-x86_64-cpython-38/kaldifeat  /alt-arabic/speech/amir/k2/tmp/kaldifeat


                make  -j4  _kaldifeat install

-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /cm/shared/apps/gcc8/8.4.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /cm/shared/apps/gcc8/8.4.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- C++ Standard version: 14
-- Downloading pybind11

  Cannot generate a safe runtime search path for target _kaldifeat because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libcufft.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcurand.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcublas.so.10] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/lib
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libnvrtc.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libnvToolsExt.so.1] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib
    runtime library [libcudart.so.10.2] in /cm/shared/apps/cuda10.2/toolkit/10.2.89/lib64 may be hidden by files in:
      /home/local/QCRI/ahussein/anaconda3/envs/k2/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  kaldifeat/python/csrc/CMakeLists.txt:2 (pybind11_add_module)


-- Generating done
-- Build files have been written to: /alt-arabic/speech/amir/k2/tmp/kaldifeat/build/temp.linux-x86_64-cpython-38
Scanning dependencies of target kaldifeat_core
[ 20%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-fbank.cc.o
[ 20%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-functions.cc.o
[ 20%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-mfcc.cc.o
[ 20%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-plp.cc.o
[ 25%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-spectrogram.cc.o
[ 30%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/feature-window.cc.o
[ 35%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/matrix-functions.cc.o
[ 40%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/mel-computations.cc.o
[ 45%] Building CXX object kaldifeat/csrc/CMakeFiles/kaldifeat_core.dir/online-feature.cc.o
[ 50%] Linking CXX shared library ../../lib/libkaldifeat_core.so
[ 50%] Built target kaldifeat_core
Scanning dependencies of target _kaldifeat
[ 70%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/feature-fbank.cc.o
[ 70%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/feature-mfcc.cc.o
[ 70%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/feature-plp.cc.o
[ 70%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/feature-spectrogram.cc.o
[ 80%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/kaldifeat.cc.o
[ 85%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/mel-computations.cc.o
[ 90%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/online-feature.cc.o
[ 95%] Building CXX object kaldifeat/python/csrc/CMakeFiles/_kaldifeat.dir/utils.cc.o
[100%] Linking CXX shared module ../../../lib/_kaldifeat.cpython-38-x86_64-linux-gnu.so
[100%] Built target _kaldifeat
[ 45%] Built target kaldifeat_core
Scanning dependencies of target test_kaldifeat
[ 90%] Built target _kaldifeat
[ 95%] Building CXX object kaldifeat/csrc/CMakeFiles/test_kaldifeat.dir/test_kaldifeat.cc.o

 File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/install_lib.py", line 112, in build
    self.run_command('build_ext')
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 84, in run

    _build_ext.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
    self.build_extensions()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 466, in build_extensions
    self._build_extensions_serial()
  File "/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 492, in _build_extensions_serial
    self.build_extension(ext)
  File "/alt-arabic/speech/amir/k2/tmp/kaldifeat/cmake/cmake_extension.py", line 124, in build_extension
    raise Exception(
Exception:
Build kaldifeat failed. Please check the error message.

CMakeOutput.log
CMakeError.log
@csukuangfj
Copy link
Owner

Could you use

git clone https://github.com/csukuangfj/kaldifeat
cd kaldifeat
mkdir build
cd build
cmake ..
make

and post the logs of make?

The above logs you posted are not informative and it is not possible to figure out what bad things happened.

@AmirHussein96
Copy link
Author

AmirHussein96 commented Nov 30, 2022 •
edited

Logs of cmake ..

-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- No CMAKE_BUILD_TYPE given, default to Release
-- C++ Standard version: 14
-- Downloading pybind11
-- pybind11 is downloaded to /alt-arabic/speech/amir/k2/tmp/kaldifeat/build/_deps/pybind11-src
-- pybind11 v2.9.2
-- Found PythonInterp: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python (found version "3.8.12")
-- Found PythonLibs: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/libpython3.8.so
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Python executable: /home/local/QCRI/ahussein/anaconda3/envs/k2/bin/python
-- TORCH_DIR: /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
CMake Warning (dev) at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:29 (find_package):
Policy CMP0074 is not set: find_package uses _ROOT variables.
Run "cmake --help-policy CMP0074" for policy details. Use the cmake_policy
command to set the policy and suppress this warning.

Environment variable CUDA_ROOT is set to:

/cm/shared/apps/cuda10.2/toolkit/10.2.89

For compatibility, CMake is ignoring the variable.
Call Stack (most recent call first):
/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
cmake/torch.cmake:14 (find_package)
CMakeLists.txt:55 (include)
This warning is for project developers. Use -Wno-dev to suppress it.

-- Found CUDA: /cm/shared/apps/cuda10.2/toolkit/10.2.89 (found version "10.2")
-- Caffe2: CUDA detected: 10.2
-- Caffe2: CUDA nvcc is: /cm/shared/apps/cuda10.2/toolkit/10.2.89/bin/nvcc
-- Caffe2: CUDA toolkit directory: /cm/shared/apps/cuda10.2/toolkit/10.2.89
-- Caffe2: Header version is: 10.2
-- Could NOT find CUDNN (missing: CUDNN_LIBRARY_PATH CUDNN_INCLUDE_PATH)
CMake Warning at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:109 (message):
Caffe2: Cannot find cuDNN library. Turning the option off
Call Stack (most recent call first):
/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
cmake/torch.cmake:14 (find_package)
CMakeLists.txt:55 (include)

-- Autodetected CUDA architecture(s): 6.0
-- Added CUDA NVCC flags for: -gencode;arch=compute_60,code=sm_60
CMake Error at /home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:96 (message):
Your installed Caffe2 version uses cuDNN but I cannot find the cuDNN
libraries. Please set the proper cuDNN prefixes and / or install cuDNN.
Call Stack (most recent call first):
/home/local/QCRI/ahussein/anaconda3/envs/k2/lib/python3.8/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:40 (find_package)
cmake/torch.cmake:14 (find_package)
CMakeLists.txt:55 (include)

-- Configuring incomplete, errors occurred!
See also "/alt-arabic/speech/amir/k2/tmp/kaldifeat/build/CMakeFiles/CMakeOutput.log".
See also "/alt-arabic/speech/amir/k2/tmp/kaldifeat/build/CMakeFiles/CMakeError.log".

@csukuangfj
Copy link
Owner

Please use

wget https://huggingface.co/csukuangfj/cudnn/resolve/main/cudnn-10.2-linux-x64-v8.0.2.39.tgz
tar xvf cudnn-10.2-linux-x64-v8.0.2.39.tgz --strip-components=1 -C /cm/shared/apps/cuda10.2/toolkit/10.2.89

to install cuDNN and retry

cd /path/to/kaldifeat
rm -rf build
mkdir build
cd build
cmake ..
make

@AmirHussein96
Copy link
Author

AmirHussein96 commented Dec 4, 2022 •
edited

Thanks @csukuangfj it worked with the following:

cmake -DCMAKE_BUILD_TYPE=Release -DCUDNN_LIBRARY_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/lib64/libcudnn.so
 -DCUDNN_INCLUDE_PATH=/cm/shared/apps/cudnn8.0-cuda10.2/8.0.5.39/include ..

make -j8

@AmirHussein96
Copy link
Author

@csukuangfj How can I export the library to use it. I tried the following but it did not work

kfeat=/alt-arabic/speech/amir/k2/tmp/kaldifeat/
export PYTHONPATH=$kfeat/kaldifeat/python:$PYTHONPATH
export PYTHONPATH=$kfeat/kaldifeat/build/lib:$PYTHONPATH

@csukuangfj
Copy link
Owner

Could you show the error logs?

@AmirHussein96
Copy link
Author

@csukuangfj there is no error, I managed to build kaldifeat from source using cmake, and make succesfully. But when I import it in python it does not work, so I though it might be similar to k2 that I need to export python path right?

@csukuangfj
Copy link
Owner

But when I import it in python it does not work,

If it does not work, it should give you some message indicating it is not working.

so I though it might be similar to k2 that I need to export python path right

Yes, if you use

git clone https://github.com/csukuangfj/kaldifeat
cd kaldifeat
mkdir build
cd build
cmake ..
make -j

Then you have to use

export PYTHONPATH=/path/to/kaldifeat/build/lib:$PYTHONPATH
export PYTHONPATH=/path/to/kaldifeat/kaldifeat/python:$PYTHONPATH

@AmirHussein96
Copy link
Author

Thanks it worked, you can close the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@csukuangfj @AmirHussein96