-
Notifications
You must be signed in to change notification settings - Fork 13
How to prepare image and build TensorFlow for ARC HSDK
*** WORK IN PROGRESS ***
TensorFlow™ is an open source software library for high performance numerical computation. This article describes 1) how to prepare microSD card image for ARC HSDK board which is capable for TensorFlow installation and 2) how to build TensorFlow and install wheel file on image. Note! All steps below which are related to Host were made on Linux host machine with Fedora 28 installed on it.
Following the instructions provided by https://crosstool-ng.github.io/docs/install/ download and install Crosstool-ng on your machine. Following the instructions provided by https://docs.bazel.build/versions/master/install.html install bazel on your machine.
This manual implies building Tensorflow as a wheel file for Python3.6. If on Linux host machine Python3.6 is not installed, follow the link: https://www.python.org/ and install latest Python3.6 version. Tensorflow is not working properly with Python3.7 yet.
Install python3 virtualenv if you do not want to pollute your python setup on host machine. In this article the tensorflow build process is going under virtualenv.
In some workflow directory run next commands:
mkdir crosstoolchain
mkdir nativetoolchain
virtualenv tensorflow_env
git clone https://github.com/EvgeniiDidin/buildroot.git
cd buildroot
git checkout br_for_tensorflow
cd ..
export WORKFLOW=${PWD}
Because of huge size of result TensorFlow binary it is necessary to build cross-toolchain with "-mlong-calls" and "-mno-millicode" flags. In crosstoolchain directory create file defconfig with next content:
CT_EXPERIMENTAL=y
CT_PREFIX_DIR="${PWD}/arc_gnu"
CT_ARCH_ARC=y
CT_ARCH_CPU="hs38"
CT_TARGET_CFLAGS="-matomic -mlong-calls -mno-millicode"
CT_TARGET_VENDOR="snps"
CT_TARGET_ALIAS="arc-linux"
CT_KERNEL_LINUX=y
CT_BINUTILS_SRC_DEVEL=y
CT_BINUTILS_DEVEL_URL="https://github.com/foss-for-synopsys-dwc-arc-processors/binutils-gdb"
CT_BINUTILS_DEVEL_BRANCH="arc-2018.09"
CT_BINUTILS_V_2_31_1=y
CT_GLIBC_SRC_DEVEL=y
CT_GLIBC_DEVEL_URL="https://github.com/foss-for-synopsys-dwc-arc-processors/glibc"
CT_GLIBC_DEVEL_BRANCH="arc-glibc-2.28"
CT_GCC_SRC_DEVEL=y
CT_GCC_DEVEL_VCS_git=y
CT_GCC_DEVEL_URL="https://github.com/foss-for-synopsys-dwc-arc-processors/gcc/"
CT_GCC_DEVEL_BRANCH="arc-2018.09"
CT_GCC_V_8_2_0=y
CT_CC_GCC_LIBQUADMATH=y
CT_CC_LANG_CXX=y
After creation of file run:
ct-ng defconfig
ct-ng build
After build successfully finishes run:
tar -zcvf arc_gnu.tar.gz arc_gnu/
Now we have proper tools for next steps.
For some packages which are required by tensorflow it is necessary to have native tools build on host. In nativetoolchain directory create file defconfig with next content:
CT_EXPERIMENTAL=y
CT_PREFIX_DIR="${PWD}/NATIVE"
CT_ARCH_ARC=y
CT_ARCH_CPU="hs38"
CT_TARGET_CFLAGS="-matomic -mlong-calls -mno-millicode"
CT_TARGET_VENDOR="snps"
CT_TARGET_ALIAS="arc-linux"
CT_CANADIAN=y
CT_HOST="arc-snps-linux-gnu"
CT_KERNEL_LINUX=y
CT_BINUTILS_SRC_DEVEL=y
CT_BINUTILS_DEVEL_URL="https://github.com/foss-for-synopsys-dwc-arc-processors/binutils-gdb"
CT_BINUTILS_DEVEL_BRANCH="arc-2018.09"
CT_BINUTILS_V_2_31_1=y
CT_GLIBC_SRC_DEVEL=y
CT_GLIBC_DEVEL_URL="https://github.com/foss-for-synopsys-dwc-arc-processors/glibc"
CT_GLIBC_DEVEL_BRANCH="arc-glibc-2.28"
CT_GCC_SRC_DEVEL=y
CT_GCC_DEVEL_VCS_git=y
CT_GCC_DEVEL_URL="https://github.com/foss-for-synopsys-dwc-arc-processors/gcc/"
CT_GCC_DEVEL_BRANCH="arc-2018.09"
CT_GCC_V_8_2_0=y
CT_CC_GCC_LIBQUADMATH=y
CT_CC_LANG_CXX=y
After creation of file run:
export PATH=$PATH:${WORKFLOW}/crosstoolchain/arc_gnu/bin
ct-ng defconfig
ct-ng build
After build ends the directory NATIVE will contain native toolchain.
To build TensorFlow package enter next commands:
cd ${WORKFLOW}/tensorflow_env
. bin/activate
pip install numpy==1.14.5
pip install keras-preprocessing
pip install keras-applications
git clone https://github.com/EvgeniiDidin/tensorflow.git
cd tensorflow
git checkout tensorflow_arc
sed -i "s@PATHTOWORKFLOW@$WORKFLOW@g" tensorflow/workspace.bzl
bazel build --sandbox_debug -c opt --cpu=arc --crosstool_top=@local_config_arc_compiler//:toolchain --verbose_failures --build_python_zip //tensorflow/tools/pip_package:build_pip_package
During build process there will be some errors:
- With high probability sha256 of crosstoolchain archive will not be correct. If error occurs replace wanted hash value with was value in tensorflow/workspace.bzl file and rerun build command. When build finishes run next commands:
./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
cp /tmp/tensorflow_pkg/tensorflow-1.10.0-cp36-cp36m-linux_x86_64.whl ${WORKFLOW}/tensorflow-1.10.0-cp36-none-any.whl
To build microSD card image enter next commands:
cd ${WORKFLOW}/buildroot
make snps_archs38_hsdk_defconfig
make
Building image will take some time. After it ends deploy prepared sdcard.img to microSD card with next commands:
cd output/images
umount /run/media/<username>/*
sudo dd if=sdcard.img of=/dev/mmcblk0 bs=4M
Using diskmanager like gparted increase to maximum size of Ext4 partition.
Tensorflow requires hdf5, h5py and grpcio, which will be build natively on board.
cd ${WORKFLOW}
wget -O hdf5-1.10.5.tar.gz "https://www.hdfgroup.org/package/hdf5-1-10-5-tar-gz/?wpdmdl=13571&refresh=5c7d2846249701551706182"
tar -xvf hdf5-1.10.5.tar.gz
wget https://files.pythonhosted.org/packages/90/2e/f432e69f86961393c22a6c4490aafdbb94533deba33f5ea2d5651770aa21/grpcio-1.18.0.tar.gz
tar -xvf grpcio-1.18.0.tar.gz
sudo cp -r ${WORKFLOW}/hdf5-1.10.5 /<<<path-to-sdcard-ext4-partition>>>/root
sudo cp -r ${WORKFLOW}/grpcio-1.18.0 /<<<path-to-sdcard-ext4-partition>>>/root
sudo cp -r ${WORKFLOW}/nativetoolchain/NATIVE /<<<path-to-sdcard-ext4-partition>>>/opt/arc_gnu
sudo cp -r ${WORKFLOW}/buildroot/output/host/arc-buildroot-linux-gnu/sysroot/usr/include /<<<path-to-sdcard-ext4-partition>>>/usr/include
sudo cp ${WORKFLOW}/tensorflow-1.10.0-cp36-none-any.whl /<<<path-to-sdcard-ext4-partition>>>/root
sync
Unmount microSD card's partitions and insert card into HSDK board.
Following article Preparing HSDK board describes all preparations before working with HSDK board.
NOTE!! For further work of the pip utility it is necessary to establish Internet connection for HSDK board via Ethernet cable. Running udhcpc utility should give such log:
# udhcpc
udhcpc: started, v1.29.3
udhcpc: sending discover
udhcpc: sending discover
udhcpc: sending select for 10.42.0.76
udhcpc: lease of 10.42.0.76 obtained, lease time 3600
deleting routers
adding dns 10.42.0.1
First of all it is necessary to build hdf5 for further installation of h5py package. After login run next commands:
cd /opt/arc_gnu/bin
remove=arc-linux-
for i in "$remove"*;do ln -s "$i" "${i#"$remove"}";done
cd /root/hdf5
/root/hdf5-1.10.5/configure
make -j4
h5py is a python api of HDF5 package, so installed HDF5 package is required:
cd /root/hdf5-1.10.5/
find . -name *.h -exec cp {} /opt/arc_gnu/arc-snps-linux-gnu/sysroot/usr/include/ \;
cd ../hdf5
find . -name *.h -exec cp {} /opt/arc_gnu/arc-snps-linux-gnu/sysroot/usr/include/ \;
cp /root/hdf5/hl/src/.libs/libhdf5_hl.so /usr/lib
cp /root/hdf5/src/.libs/libhdf5.so /usr/lib/
pip install h5py
Modify file /root/grpcio-1.18.0/third_party/boringssl/include/openssl/base.h
diff --git a/include/openssl/base.h b/include/openssl/base.h
index 7fe232f3b..8d062eab5 100644
--- a/include/openssl/base.h
+++ b/include/openssl/base.h
@@ -114,6 +114,8 @@ extern "C" {
#define OPENSSL_32_BIT
#elif defined(__myriad2__)
#define OPENSSL_32_BIT
+#elif defined(__arc__)
+#define OPENSSL_32_BIT
#else
After that run next commands:
cd /root/grpcio-1.18.0
pip install .
To install TensorFlow enter next commands:
cd /root/
pip install tensorflow-1.10.0-cp36-none-any.whl
After installation completes tensorflow package can be used in python applications.
# python
Python 3.6.3 (default, Feb 11 2019, 15:24:16)
[GCC 8.2.1 20180814] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
>>> print(tensorflow.__version__)
1.10.0