Torchaudio Compilation Error #674

jimchen90 · 2020-06-01T15:20:12Z

🐛 Bug

When trying to compile torchaudio from source after python setup.py install within anaconda, it fails with an error that configure: error: cannot find flac (see below for full information).

To Reproduce

Steps to reproduce the behavior:

Compile PyTorch from source in conda
Compile Torchaudio from source in conda

See code as below:

module load cuda/10.1
conda create -n audio-master python=3.7
conda activate audio-master
# install cuda with needed version
conda install -c pytorch magma-cuda101

conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
git submodule sync
git submodule update --init --recursive
git submodule sync
git submodule update --init --recursive

python setup.py clean --all
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py install

cd ..
python -c "import torch; print(torch.__version__); print(torch.cuda.is_available());"

conda install -c conda-forge sox
git clone https://github.com/pytorch/audio
cd audio

python setup.py clean --all              
python setup.py install

cd ..
python -c "import torchaudio;"

Error message:

After python setup.py install for torchaudio, it shows:

Building sox
configure: error: in `/private/home/jimchen90/audio/third_party/tmp/sox-14.4.2':
configure: error: cannot find flac
See `config.log' for more details
subprocess.CalledProcessError: Command '['/private/home/jimchen90/audio/build_tools/setup_helpers/build_third_party.sh']' returned non-zero exit status 1.

Expected behavior

Torchaudio can be imported outside the source folder.

Environment

PyTorch Version: 1.6.0a0+78acc9d
OS: Linux
How you installed PyTorch: Conda
Build command you used : As described above
Python version: 3.7
CUDA/cuDNN version: CUDA 10.1

Additional context

With the help of Moto @mthrok , torchaudio can be imported after adding an external sox file in the audio directory by:

conda uninstall sox
rm -rf third_party
touch .use_external_sox
python setup.py clean develop

The result of python -m torch.utils.collect_env

Collecting environment information...
PyTorch version: 1.6.0a0+78acc9d
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.14.0

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.1.105
GPU models and configuration:
GPU 0: Quadro GP100
GPU 1: Quadro GP100

Nvidia driver version: 418.116.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.6.0a0+78acc9d
[pip] torchaudio==0.6.0a0+fd73d8f
[conda] blas                      1.0                         mkl
[conda] magma-cuda101             2.5.2                         1    pytorch
[conda] mkl                       2020.1                      217
[conda] mkl-include               2020.1                      217
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.15           py37ha843d7b_0
[conda] mkl_random                1.1.1            py37h0573a6f_0
[conda] numpy                     1.18.1           py37h4f9e942_0
[conda] numpy-base                1.18.1           py37hde5b4d6_1
[conda] torch                     1.6.0a0+78acc9d          pypi_0    pypi
[conda] torchaudio                0.6.0a0+fd73d8f           dev_0    <develop>

Related issues can be seen in #666 and #669. Thank @vincentqb and @mthrok for helping me solve this issue.

The text was updated successfully, but these errors were encountered:

mthrok · 2020-06-01T15:41:42Z

@jimchen90 Can you paste the result of python -m torch.utils.collect_env?

jimchen90 · 2020-06-02T01:14:34Z

@jimchen90 Can you paste the result of python -m torch.utils.collect_env?

Hi @mthrok , I paste the result of python -m torch.utils.collect_env in Additional Context. Thanks.

cpuhrsch · 2020-06-02T19:17:00Z

Maybe try installing flac with apt-get install flac

mthrok · 2020-06-16T00:48:35Z

I looked into this and the cause was that libogg present in the system when compiling flac, but not compiling sox.

In my case, libogg was present in /usr/lib/x86_64-linux-gnu

$ find /usr/lib/ -type f -name 'libogg*'
/usr/lib/x86_64-linux-gnu/libogg.a
/usr/lib/x86_64-linux-gnu/libogg.so.0.8.2

and when building with BUILD_SOX=1 python setup.py develop, flac finds it and enables it with wrong directory.

third_party/tmp/flac-1.3.2/config.log

...
configure:19217: checking for Ogg
configure:19252: gcc -o conftest -fPIC -I/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/include   conftest.c -lm  -L/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib -logg >&5
configure:19252: $? = 0
configure:19252: ./conftest
configure:19252: $? = 0
...
FLAC__HAS_OGG='1'
...
FLaC__HAS_OGG_FALSE='#'
FLaC__HAS_OGG_TRUE=''
...
OGG_CFLAGS='-I/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/include'
OGG_LIBS='-L/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib -logg'
OGG_PACKAGE='ogg'
...
#define FLAC__HAS_OGG 1

but libogg does not exist in third_party/build directory, so the above flag is wrong.

When sox is being built and when sox checks flac usability, it fails to find libogg thus it fails with error regarding flac.

third_party/tmp/sox-14.4.2/config.log

...
configure:14871: checking FLAC/all.h usability
configure:14871: gcc -c -fPIC -fstack-protector -Wall -W -Wmissing-prototypes -Wstrict-prototypes -pedantic -fopenmp -I/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/include conftest.c >&5
configure:14871: $? = 0
configure:14871: result: yes
configure:14871: checking FLAC/all.h presence
configure:14871: gcc -E -I/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/include conftest.c
configure:14871: $? = 0
configure:14871: result: yes
configure:14871: checking for FLAC/all.h
configure:14871: result: yes
configure:14873: checking whether FLAC_API_VERSION_CURRENT is declared
configure:14873: gcc -c -fPIC -fstack-protector -Wall -W -Wmissing-prototypes -Wstrict-prototypes -pedantic -fopenmp -I/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/include conftest.c >&5
conftest.c:53:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
 main ()
 ^~~~
configure:14873: $? = 0
configure:14873: result: yes
configure:14876: checking for FLAC__stream_encoder_new in -lFLAC
configure:14901: gcc -o conftest -fPIC -fstack-protector -Wall -W -Wmissing-prototypes -Wstrict-prototypes -pedantic -fopenmp -I/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/include -L/scratch/moto/torchaudio\
/build_tools/setup_helpers/../../third_party/build/lib -Wl,-z,defs conftest.c -lFLAC   -lm  >&5
conftest.c:57:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
 char FLAC__stream_encoder_new ();
 ^~~~
conftest.c:59:1: warning: function declaration isn't a prototype [-Wstrict-prototypes]
 main ()
 ^~~~
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_encoder_aspect.o): In function `FLAC__ogg_encoder_aspect_init':
ogg_encoder_aspect.c:(.text+0xb): undefined reference to `ogg_stream_init'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_encoder_aspect.o): In function `FLAC__ogg_encoder_aspect_write_callback_wrapper':
ogg_encoder_aspect.c:(.text+0x1af): undefined reference to `ogg_stream_packetin'
ogg_encoder_aspect.c:(.text+0x20c): undefined reference to `ogg_stream_flush'
ogg_encoder_aspect.c:(.text+0x268): undefined reference to `ogg_stream_pageout'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_encoder_aspect.o): In function `FLAC__ogg_encoder_aspect_finish':
ogg_encoder_aspect.c:(.text+0x45): undefined reference to `ogg_stream_clear'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_helper.o): In function `simple_ogg_page__get_at':
ogg_helper.c:(.text+0x3e2): undefined reference to `ogg_page_checksum_set'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_helper.o): In function `simple_ogg_page__set_at':
ogg_helper.c:(.text+0x464): undefined reference to `ogg_page_checksum_set'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_decoder_aspect.o): In function `FLAC__ogg_decoder_aspect_init':
ogg_decoder_aspect.c:(.text+0xc): undefined reference to `ogg_stream_init'
ogg_decoder_aspect.c:(.text+0x28): undefined reference to `ogg_sync_init'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_decoder_aspect.o): In function `FLAC__ogg_decoder_aspect_finish':
ogg_decoder_aspect.c:(.text+0x6c): undefined reference to `ogg_sync_clear'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_decoder_aspect.o): In function `FLAC__ogg_decoder_aspect_flush':
ogg_decoder_aspect.c:(.text+0xa9): undefined reference to `ogg_stream_reset'
ogg_decoder_aspect.c:(.text+0xb5): undefined reference to `ogg_sync_reset'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_decoder_aspect.o): In function `FLAC__ogg_decoder_aspect_reset':
ogg_decoder_aspect.c:(.text+0xd9): undefined reference to `ogg_stream_reset'
ogg_decoder_aspect.c:(.text+0xe5): undefined reference to `ogg_sync_reset'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_decoder_aspect.o): In function `FLAC__ogg_decoder_aspect_read_callback_wrapper':
ogg_decoder_aspect.c:(.text+0x20d): undefined reference to `ogg_sync_pageout'
ogg_decoder_aspect.c:(.text+0x231): undefined reference to `ogg_stream_pagein'
ogg_decoder_aspect.c:(.text+0x274): undefined reference to `ogg_sync_wrote'
ogg_decoder_aspect.c:(.text+0x2cc): undefined reference to `ogg_stream_packetout'
ogg_decoder_aspect.c:(.text+0x393): undefined reference to `ogg_sync_buffer'
ogg_decoder_aspect.c:(.text+0x414): undefined reference to `ogg_page_serialno'
/scratch/moto/torchaudio/build_tools/setup_helpers/../../third_party/build/lib/libFLAC.a(ogg_decoder_aspect.o): In function `FLAC__ogg_decoder_aspect_finish':
ogg_decoder_aspect.c:(.text+0x76): undefined reference to `ogg_stream_clear'
collect2: error: ld returned 1 exit status
configure:14901: $? = 1

mthrok · 2020-06-16T00:50:04Z

Adding --disable-ogg in build_flac function resolved this.

mthrok · 2020-06-16T00:59:47Z

The above solution led to another issue with

test/test_sox_io.py:7: in <module>
    import torchaudio
torchaudio/__init__.py:1: in <module>
    from .extension import (
torchaudio/extension/__init__.py:7: in <module>
    _init_extension()
torchaudio/extension/extension.py:11: in _init_extension
    _init_script_module(ext)
torchaudio/extension/extension.py:18: in _init_script_module
    torch.classes.load_library(path)
../pytorch/torch/_classes.py:46: in load_library
    torch.ops.load_library(path)
../pytorch/torch/_ops.py:105: in load_library
    ctypes.CDLL(path)
/home/moto/conda/envs/PY3.8-cuda101/lib/python3.8/ctypes/__init__.py:373: in __init__
    self._handle = _dlopen(self._name, mode)
E   OSError: /scratch/moto/torchaudio/torchaudio/_torchaudio.so: undefined symbol: op_pcm_seek

This was caused by the fact opusfile found in /usr/include/opus / /usr/lib/libopusfile.

configure:14728: checking for OPUS
configure:14735: $PKG_CONFIG --exists --print-errors "opusfile"
configure:14738: $? = 0
configure:14752: $PKG_CONFIG --exists --print-errors "opusfile"
configure:14755: $? = 0
...
pkg_cv_OPUS_CFLAGS=-I/usr/include/opus
pkg_cv_OPUS_LIBS=-lopusfile

Similarly, adding --with-opus=no to build_sox function resolved this.

mthrok · 2020-07-02T14:09:22Z

fixed by #750 and #755

Remove numbers from tutorial names

mthrok added the build label Jun 3, 2020

mthrok closed this as completed Jun 16, 2020

mthrok reopened this Jun 16, 2020

mthrok mentioned this issue Jun 26, 2020

Add ogg/vorbis support to binary distribution #750

Merged

mthrok closed this as completed Jul 2, 2020

mthrok pushed a commit to mthrok/audio that referenced this issue Feb 26, 2021

Merge pull request pytorch#674 from pytorch/remove_tutorial_numbers

ea34cb0

Remove numbers from tutorial names

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Torchaudio Compilation Error #674

Torchaudio Compilation Error #674

jimchen90 commented Jun 1, 2020 •

edited

mthrok commented Jun 1, 2020

jimchen90 commented Jun 2, 2020

cpuhrsch commented Jun 2, 2020 •

edited

mthrok commented Jun 16, 2020

mthrok commented Jun 16, 2020

mthrok commented Jun 16, 2020

mthrok commented Jul 2, 2020

Torchaudio Compilation Error #674

Torchaudio Compilation Error #674

Comments

jimchen90 commented Jun 1, 2020 • edited

🐛 Bug

To Reproduce

Expected behavior

Environment

Additional context

mthrok commented Jun 1, 2020

jimchen90 commented Jun 2, 2020

cpuhrsch commented Jun 2, 2020 • edited

mthrok commented Jun 16, 2020

mthrok commented Jun 16, 2020

mthrok commented Jun 16, 2020

mthrok commented Jul 2, 2020

jimchen90 commented Jun 1, 2020 •

edited

cpuhrsch commented Jun 2, 2020 •

edited