Failed to build OpenBLAS? #122

Qvodman · 2018-01-17T01:43:51Z

make: *** [getarch_2nd] Error 1
make -j 1 -f Makefile.install install
make[1]: Entering directory '/caffe-android-lib/OpenBLAS'
Generating openblas_config.h in /caffe-android-lib/android_lib/openblas/include
Generating f77blas.h in /caffe-android-lib/android_lib/openblas/include
Generating cblas.h in /caffe-android-lib/android_lib/openblas/include
Copying the static library to /caffe-android-lib/android_lib/openblas/lib
install: cannot stat 'libopenblas_atom-r0.2.18.a': No such file or directory
make[1]: *** [install] Error 1
make: *** [install] Error 2
Makefile.install:23: recipe for target 'install' failed
make[1]: Leaving directory '/caffe-android-lib/OpenBLAS'
Makefile:313: recipe for target 'install' failed
Failed to build OpenBLAS

sh1r0 · 2018-01-17T07:32:55Z

Regarding build issues, please try to use docker for simplicity.

Qvodman · 2018-01-17T07:57:10Z

@sh1r0 I already use docker.

sh1r0 · 2018-01-17T08:00:45Z

@Qvodman Could you show me the command you use?

Qvodman · 2018-01-17T08:03:31Z

@sh1r0
git clone --recursive https://github.com/sh1r0/caffe-android-lib.git

cd caffe-android-lib

sudo docker build -t caffe-android-lib .

sudo docker run --rm --name caffe-android-builder -e ANDROID_ABI=x86_64 -e N_JOBS=2 -v $(pwd)/android_lib/x86_64:/caffe-android-lib/android_lib caffe-android-lib ./build.sh

Qvodman · 2018-01-17T08:09:28Z

@sh1r0 And when I used:

git clone --recursive https://github.com/sh1r0/caffe-android-lib.git

cd caffe-android-lib

sudo docker build -t caffe-android-lib .

sudo docker run --rm --name caffe-android-builder -e ANDROID_ABI=arm64-v8a -e N_JOBS=2 -v $(pwd)/android_lib/arm64-v8a:/caffe-android-lib/android_lib caffe-android-lib ./build.sh

The problem:
make: *** [getarch_2nd] Error 1
make -j 1 -f Makefile.install install
make[1]: Entering directory '/caffe-android-lib/OpenBLAS'
Generating openblas_config.h in /caffe-android-lib/android_lib/openblas/include
Generating f77blas.h in /caffe-android-lib/android_lib/openblas/include
Generating cblas.h in /caffe-android-lib/android_lib/openblas/include
Copying the static library to /caffe-android-lib/android_lib/openblas/lib
install: cannot stat 'libopenblas_armv8-r0.2.18.a': No such file or directory
make[1]: *** [install] Error 1
make: *** [install] Error 2
Makefile.install:23: recipe for target 'install' failed
make[1]: Leaving directory '/caffe-android-lib/OpenBLAS'
Makefile:313: recipe for target 'install' failed
Failed to build OpenBLAS

sh1r0 · 2018-01-17T08:32:09Z

Hi @Qvodman, I just did a clean clone, re-created the docker image, and used the command below.

docker run --rm --name caffe-android-builder \
     -e ANDROID_ABI=arm64-v8a \
     -e N_JOBS=2 \
     -v $(pwd)/android_lib/arm64-v8a:/caffe-android-lib/android_lib \
     caffe-android-lib ./build.sh

Per log (listed below), OpenBLAS is successfully built and installed.

make -j 4 -f Makefile.install install
make[1]: Entering directory '/caffe-android-lib/OpenBLAS'
Generating openblas_config.h in /caffe-android-lib/android_lib/openblas/include
Generating f77blas.h in /caffe-android-lib/android_lib/openblas/include
Generating cblas.h in /caffe-android-lib/android_lib/openblas/include
Copying the static library to /caffe-android-lib/android_lib/openblas/lib
Copying the shared library to /caffe-android-lib/android_lib/openblas/lib
Generating OpenBLASConfig.cmake in /caffe-android-lib/android_lib/openblas/lib/cmake/openblas
Generating OpenBLASConfigVersion.cmake in /caffe-android-lib/android_lib/openblas/lib/cmake/openblas
Install OK!
make[1]: Leaving directory '/caffe-android-lib/OpenBLAS'

Could you provide more details about your environment?

Qvodman · 2018-01-17T08:43:47Z

@sh1r0
Ubuntu 16.04.02
cmake version 3.5.1
NDK:android-ndk-r11C

sh1r0 · 2018-01-17T09:23:47Z

Hi @Qvodman, I pulled the latest ubuntu:16.04 docker image and tested again without failures. Would you like to re-pull the docker image?

Qvodman · 2018-01-17T09:27:21Z

@sh1r0 Do I need to install OpenBLAS by myself?

sh1r0 · 2018-01-17T09:28:52Z

Nope, everything should be self-contained in this project.

Qvodman · 2018-01-17T09:30:58Z

liaolp@ubuntu:~/caffe-android-lib$ sudo docker build -t caffe-android-lib .
Sending build context to Docker daemon 309.9 MB
Step 1/5 : FROM ubuntu:16.04
 ---> 00fd29ccc6f1
Step 2/5 : RUN apt-get update && apt-get install -y --no-install-recommends     automake     build-essential     ca-certificates     curl     cmake     file     libtool     pkg-config     unzip     wget
 ---> Using cache
 ---> e21d0c6626ad
Step 3/5 : RUN curl -SL     http://dl.google.com/android/repository/android-ndk-r11c-linux-x86_64.zip     -o /tmp/android-ndk.zip     && unzip -q -d /opt /tmp/android-ndk.zip     && rm -f /tmp/android-ndk.zip
 ---> Using cache
 ---> 78131cbb08f3
Step 4/5 : COPY . /caffe-android-lib
 ---> Using cache
 ---> 081e19e42ff4
Step 5/5 : WORKDIR /caffe-android-lib
 ---> Using cache
 ---> 3c628ed3e218
Successfully built 3c628ed3e218

sh1r0 · 2018-01-17T09:38:15Z

First, docker pull ubuntu:16.04 to get the latest image, which image id should be 2a4cca5ac898.
Then, re-build image via docker build -t caffe-android-lib ..

Qvodman · 2018-01-17T10:38:24Z

sudo docker images
[sudo] password for liaolp: 
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
caffe-android-lib   latest              80ac207e9df6        21 seconds ago      3.45 GB
<none>              <none>              ea38e441a68f        2 hours ago         3.45 GB
<none>              <none>              1038af21e2e6        8 hours ago         3.45 GB
ubuntu              16.04               2a4cca5ac898        37 hours ago        111 MB
<none>              <none>              3c628ed3e218        11 days ago         3.45 GB
<none>              <none>              e29454a39e35        12 days ago         3.46 GB
ubuntu              <none>              00fd29ccc6f1        4 weeks ago         111 MB

sh1r0 · 2018-01-17T13:40:25Z

According to your comment above (quoted below),

Step 1/5 : FROM ubuntu:16.04
---> 00fd29ccc6f1

I think the caffe-android-lib image you created is not based on the right one

ubuntu              16.04               2a4cca5ac898        37 hours ago        111 MB

but this one

ubuntu              <none>              00fd29ccc6f1        4 weeks ago         111 MB

Qvodman · 2018-01-18T01:12:05Z

@sh1r0 I re-created the docker image

sudo docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
caffe-android-lib   latest              7383e0d3c6ef        8 minutes ago       3.47 GB
ubuntu              16.04               2a4cca5ac898        2 days ago          111 MB

But the problem is still reproduced：

make: *** [getarch_2nd] Error 1
make -j 1 -f Makefile.install install
make[1]: Entering directory '/caffe-android-lib/OpenBLAS'
Generating openblas_config.h in /caffe-android-lib/android_lib/openblas/include
Generating f77blas.h in /caffe-android-lib/android_lib/openblas/include
Generating cblas.h in /caffe-android-lib/android_lib/openblas/include
Copying the static library to /caffe-android-lib/android_lib/openblas/lib
install: cannot stat 'libopenblas_armv8-r0.2.18.a': No such file or directory
make[1]: *** [install] Error 1
Makefile.install:23: recipe for target 'install' failed
make[1]: Leaving directory '/caffe-android-lib/OpenBLAS'
Makefile:313: recipe for target 'install' failed
Failed to build OpenBLAS
make: *** [install] Error 2

sh1r0 · 2018-01-18T01:32:36Z

Could you check the version of cmake via docker run --rm --name caffe-android-builder caffe-android-lib cmake --version?

Qvodman · 2018-01-18T01:33:51Z

sudo docker run --rm --name caffe-android-builder caffe-android-lib cmake --version
cmake version 3.5.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

sh1r0 · 2018-01-18T02:19:39Z

Hi @Qvodman, I got another linux machine to test from scratch without any issues. I think the docker build should be fine on a clean setup. Sorry, I could not help you to find out why this did not work on your env.

Qvodman · 2018-01-18T09:24:42Z

@sh1r0 The new issues:

/home/liaolp/caffe-android-lib/android_lib/opencv/sdk/native/jni/include/opencv2/core/mat.hpp:1754:33: note:   template argument deduction/substitution failed:
/home/liaolp/caffe-android-lib/android_lib/opencv/sdk/native/jni/include/opencv2/core/mat.hpp:1759:39: note: template<class _Tp> const _Tp& cv::Mat::at(cv::Point) const
    template<typename _Tp> const _Tp& at(Point pt) const;
                                      ^
/home/liaolp/caffe-android-lib/android_lib/opencv/sdk/native/jni/include/opencv2/core/mat.hpp:1759:39: note:   template argument deduction/substitution failed:
android/CMakeFiles/caffe_jni.dir/build.make:86: recipe for target 'android/CMakeFiles/caffe_jni.dir/caffe_mobile.cpp.o' failed
make[2]: *** [android/CMakeFiles/caffe_jni.dir/caffe_mobile.cpp.o] Error 1
CMakeFiles/Makefile2:1053: recipe for target 'android/CMakeFiles/caffe_jni.dir/all' failed
make[1]: *** [android/CMakeFiles/caffe_jni.dir/all] Error 2
Makefile:127: recipe for target 'all' failed
make: *** [all] Error 2

mmiirroo · 2018-04-18T10:35:51Z

same problem

make[1]:
Leaving directory '/caffe-android-lib/OpenBLAS/kernel'
make -j 1 -f Makefile.install install
make[1]: Entering directory '/caffe-android-lib/OpenBLAS'
Generating openblas_config.h in /caffe-android-lib/android_lib/openblas/include
Generating f77blas.h in /caffe-android-lib/android_lib/openblas/include
Generating cblas.h in /caffe-android-lib/android_lib/openblas/include
Copying the static library to /caffe-android-lib/android_lib/openblas/lib
install: cannot stat 'libopenblas_atom-r0.2.18.a': No such file or directory
Makefile.install:23: recipe for target 'install' failed
make[1]: *** [install] Error 1
make[1]: Leaving directory '/caffe-android-lib/OpenBLAS'
Makefile:313: recipe for target 'install' failed
make: *** [install] Error 2
Failed to build OpenBLAS

and compile command is the following:

echo Copying the static library to /opt/OpenBLAS/lib
install -pm644 libopenblas_atom-r0.2.18.a /opt/OpenBLAS/lib
cd /opt/OpenBLAS/lib ;
ln -fs libopenblas_atom-r0.2.18.a libopenblas.a

sh1r0 · 2018-04-19T09:33:20Z

Hi @Qvodman and @seiyaa,
I've used docker to build the project on windows 10 and ubuntu 14.04 without any problems. FYI, I list the openblas libs I got for arm64-v8a, x86_64, and x86, respectively. Note that the filename of all .a has a p following the arch name, which is different from yours.

android_lib/arm64-v8a/openblas/lib/:
cmake  libopenblas.a  libopenblas_armv8p-r0.2.18.a

android_lib/x86_64/openblas/lib/:
cmake  libopenblas.a  libopenblas_atomp-r0.2.18.a

android_lib/x86/openblas/lib/:
cmake  libopenblas.a  libopenblas_atomp-r0.2.18.a

adityakamble49 · 2018-04-19T10:06:26Z

I am facing the same issue
Getting following error

make[1]: Leaving directory `/home/aditya/workspace/caffe-android-lib/OpenBLAS/kernel'
make -j 1 -f Makefile.install install
make[1]: Entering directory `/home/aditya/workspace/caffe-android-lib/OpenBLAS'
Generating openblas_config.h in /home/aditya/workspace/caffe-android-lib/android_lib/openblas/include
Generating f77blas.h in /home/aditya/workspace/caffe-android-lib/android_lib/openblas/include
Generating cblas.h in /home/aditya/workspace/caffe-android-lib/android_lib/openblas/include
Copying the static library to /home/aditya/workspace/caffe-android-lib/android_lib/openblas/lib
install: cannot stat ‘libopenblas_armv8-r0.2.18.a’: No such file or directory
make[1]: *** [install] Error 1
make[1]: Leaving directory `/home/aditya/workspace/caffe-android-lib/OpenBLAS'
make: *** [install] Error 2
Failed to build OpenBLAS

As you mentioned p following arch name, there is same file generated under OpenBLAS directory and linked to libopenblas.a also . So why script expecting without p following arch name in libopenblas file name.

caffe-android-lib $ ls -l OpenBLAS | grep libopenblas
lrwxrwxrwx  1 aditya aditya      28 Apr 19 15:26 libopenblas.a -> libopenblas_armv8p-r0.2.18.a
-rw-rw-r--  1 aditya aditya 4754848 Apr 19 15:29 libopenblas_armv8p-r0.2.18.a

I have used following environment to build

caffe-android-lib $ uname -a
Linux ubuntu 4.4.0-31-generic #50~14.04.1-Ubuntu SMP Wed Jul 13 01:07:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
caffe-android-lib $ cmake --version
cmake version 3.5.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

and android-ndk-r11c for android ndk

sh1r0 · 2018-04-19T10:09:01Z

Hi @adityakamble49, could you have a try on docker build?

adityakamble49 · 2018-04-19T10:10:41Z

@sh1r0 , Yes tried docker build , giving same error

sh1r0 · 2018-04-19T10:13:14Z

@adityakamble49 Could you let me know your docker version (via docker --version)?

adityakamble49 · 2018-04-19T10:16:17Z

@sh1r0 Sure, Docker version as below

caffe-android-lib $ docker --version
Docker version 18.04.0-ce, build 3d479c0

adityakamble49 · 2018-04-20T06:42:11Z

@sh1r0 I Got openblas builds from https://github.com/solrex/caffe-mobile

Copied it in the repository

caffe-android-lib $ tree android_lib/
android_lib/
├── bin
├── include
│   ├── cblas.h
│   ├── f77blas.h
│   ├── lapacke_config.h
│   ├── lapacke.h
│   ├── lapacke_mangling.h
│   ├── lapacke_utils.h
│   └── openblas_config.h
└── lib
    ├── cmake
    │   └── openblas
    │       ├── OpenBLASConfig.cmake
    │       └── OpenBLASConfigVersion.cmake
    ├── libopenblas.a -> libopenblas_armv8p-r0.2.19.a
    └── libopenblas_armv8p-r0.2.19.a

5 directories, 11 files

Commented build_openblas.sh in build.sh

caffe-android-lib $ git diff
diff --git a/build.sh b/build.sh
index 6f8c954..552ab11 100755
--- a/build.sh
+++ b/build.sh
@@ -7,10 +7,10 @@ set -eu

 pushd "${PROJECT_DIR}"

-if ! ./scripts/build_openblas.sh; then
-    echo "Failed to build OpenBLAS"
-    exit 1
-fi
+#if ! ./scripts/build_openblas.sh; then
+#    echo "Failed to build OpenBLAS"
+#    exit 1
+#fi

 ./scripts/build_boost.sh
 ./scripts/build_gflags.sh

Able to build final libcaffe_jni.so and libcaffe.so

caffe-android-lib $ tree android_lib/caffe/lib/
android_lib/caffe/lib/
├── libcaffe_jni.so
├── libcaffe.so
└── libproto.a

0 directories, 3 files

luohl364218 · 2018-10-18T12:27:29Z

@adityakamble49 your solution works!

Bshowg · 2018-11-28T13:21:58Z

Disabling multi-threading in scripts/openblas.sh makes the error disappear and the library builds.
In particular change line 59 of openblash.sh to :
HOSTCC= gcc USE_THREAD=0 NUM_THREADS=1 USE_OPENMP=0

Although, the process is really slow. Any idea why this way it works?

EDIT: using the docker

sh1r0 closed this as completed Jan 17, 2018

sh1r0 added the duplicate label Jan 17, 2018

sh1r0 reopened this Jan 17, 2018

sh1r0 removed the duplicate label Jan 17, 2018

adityakamble49 mentioned this issue Dec 11, 2018

Fixed OpenBLAS builds #132

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to build OpenBLAS? #122

Failed to build OpenBLAS? #122

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018

Qvodman commented Jan 17, 2018

Qvodman commented Jan 17, 2018 •

edited

Loading

sh1r0 commented Jan 17, 2018 •

edited

Loading

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018 •

edited

Loading

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018

Qvodman commented Jan 17, 2018 •

edited

Loading

sh1r0 commented Jan 17, 2018 •

edited

Loading

Qvodman commented Jan 18, 2018

sh1r0 commented Jan 18, 2018

Qvodman commented Jan 18, 2018

sh1r0 commented Jan 18, 2018

Qvodman commented Jan 18, 2018

mmiirroo commented Apr 18, 2018

sh1r0 commented Apr 19, 2018

adityakamble49 commented Apr 19, 2018

sh1r0 commented Apr 19, 2018

adityakamble49 commented Apr 19, 2018

sh1r0 commented Apr 19, 2018

adityakamble49 commented Apr 19, 2018

adityakamble49 commented Apr 20, 2018

luohl364218 commented Oct 18, 2018

Bshowg commented Nov 28, 2018 •

edited

Loading

Failed to build OpenBLAS? #122

Failed to build OpenBLAS? #122

Comments

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018

Qvodman commented Jan 17, 2018

Qvodman commented Jan 17, 2018 • edited Loading

sh1r0 commented Jan 17, 2018 • edited Loading

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018 • edited Loading

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018

Qvodman commented Jan 17, 2018

sh1r0 commented Jan 17, 2018

Qvodman commented Jan 17, 2018 • edited Loading

sh1r0 commented Jan 17, 2018 • edited Loading

Qvodman commented Jan 18, 2018

sh1r0 commented Jan 18, 2018

Qvodman commented Jan 18, 2018

sh1r0 commented Jan 18, 2018

Qvodman commented Jan 18, 2018

mmiirroo commented Apr 18, 2018

sh1r0 commented Apr 19, 2018

adityakamble49 commented Apr 19, 2018

sh1r0 commented Apr 19, 2018

adityakamble49 commented Apr 19, 2018

sh1r0 commented Apr 19, 2018

adityakamble49 commented Apr 19, 2018

adityakamble49 commented Apr 20, 2018

luohl364218 commented Oct 18, 2018

Bshowg commented Nov 28, 2018 • edited Loading

Qvodman commented Jan 17, 2018 •

edited

Loading

sh1r0 commented Jan 17, 2018 •

edited

Loading

sh1r0 commented Jan 17, 2018 •

edited

Loading

Qvodman commented Jan 17, 2018 •

edited

Loading

sh1r0 commented Jan 17, 2018 •

edited

Loading

Bshowg commented Nov 28, 2018 •

edited

Loading