enable android build #715

daquexian · 2019-03-27T04:46:31Z

Fix #271, tested with android ndk r19b.

I don't use build.sh but only cmake

cmake -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK/build/cmake/android.toolchain.cmake -DANDROID_CPP_FEATURES=exceptions -DANDROID_PLATFORM=android-28 -DANDROID_ABI=arm64-v8a -DCMAKE_BUILD_TYPE=Release -Donnxruntime_CROSS_COMPILING=ON -Donnxruntime_BUILD_x86=OFF -DONNX_CUSTOM_PROTOC_EXECUTABLE=/usr/bin/protoc ../cmake

My changes:

Android NDK (at least for r19b) always output x86_64-unknown-linux-gnu for -dumpmachine on my PC, so the arch should be determined by the cmake official way
Android NDK doesn't support filesystem so far (std::filesystem support android/ndk#609), so I set the two flags explicitly for Android. What's more, the header_files_test should be disabled if filesystem is not supported. Filesystem is not needed since fb2a44f.

The result of onnxruntime_test_all on Android:

[----------] Global test environment tear-down
[==========] 957 tests from 104 test cases ran. (3713 ms total)
[ PASSED ] 918 tests.
[ FAILED ] 39 tests, listed below:
[ FAILED ] GraphTransformationTests.IdentityElimination
[ FAILED ] GraphTransformationTests.SliceElimination
[ FAILED ] GraphTransformationTests.ConstantFolding1
[ FAILED ] GraphTransformationTests.FuseConvBNMulAddUnsqueeze
[ FAILED ] GraphTransformationTests.FuseConvActivation
[ FAILED ] GraphTransformationTests.FuseConvBNNoBias
[ FAILED ] GraphTransformationTests.FuseConvMulNoBias
[ FAILED ] GraphTransformationTests.FuseConvAddNoBias
[ FAILED ] GraphTransformationTests.FuseConvBNMulAddUnsqueezeNoBias
[ FAILED ] GraphTransformationTests.FuseConvAddMul3D
[ FAILED ] GraphTransformationTests.MatMulAddFusion_two_input
[ FAILED ] GraphTransformationTests.MatMulAddFusion_three_input
[ FAILED ] GraphTransformationTests.Gemm_Relu_three_input
[ FAILED ] GraphTransformationTests.FuseConvBnAddMulFloat16
[ FAILED ] Float16_Tests.Mul_16_Test
[ FAILED ] InferenceSessionTests.NoTimeout
[ FAILED ] InferenceSessionTests.DisableCPUArena
[ FAILED ] InferenceSessionTests.CheckRunLogger
[ FAILED ] InferenceSessionTests.CheckRunProfilerWithSessionOptions
[ FAILED ] InferenceSessionTests.CheckRunProfilerWithStartProfile
[ FAILED ] InferenceSessionTests.MultipleSessionsNoTimeout
[ FAILED ] InferenceSessionTests.PreAllocateOutputVector
[ FAILED ] InferenceSessionTests.ConfigureVerbosityLevel
[ FAILED ] InferenceSessionTests.TestWithIstream
[ FAILED ] InferenceSessionTests.TestRegisterExecutionProvider
[ FAILED ] InferenceSessionTests.InvalidInputTypeOfTensorElement
[ FAILED ] InferenceSessionTests.ModelWithoutOpset
[ FAILED ] InferenceSessionTests.TestTruncatedSequence
[ FAILED ] InferenceSessionTests.TestCopyToFromDevices
[ FAILED ] InferenceSessionTests.TestL1Transformers
[ FAILED ] InferenceSessionTests.TestL1AndL2Transformers
[ FAILED ] InferenceSessionTests.TestCustomTransformers
[ FAILED ] InferenceSessionTests.DisableAllTransformers
[ FAILED ] CustomKernelTests.CustomKernelWithBuildInSchema
[ FAILED ] CustomKernelTests.CustomKernelWithCustomSchema
[ FAILED ] CustomKernelTests.CustomKernelWithOptionalOutput
[ FAILED ] MemcpyTest.copy1
[ FAILED ] Random.MultinomialGoodCase
[ FAILED ] Random.MultinomialDefaultDType

msftclas · 2019-03-27T04:46:42Z

All CLA requirements met.

snnn · 2019-03-27T08:51:47Z

libprotobuf-lite is a static lib, you don't need to call target_link_libraries on it. Therefore, I don't think you need to change protobuf's source code.

cmake/onnxruntime_unittests.cmake

daquexian · 2019-03-27T11:43:27Z

@snnn Thanks for your review. Even though protobuf-lite is a static lib, the compiler flag will be spread to the executable by cmake.

And what is your opinion on protocolbuffers/protobuf#2719? -llog should be added somewhere to solve the undefined reference.

daquexian · 2019-03-30T12:30:45Z

@snnn I have rebased this branch on master. However I believe -llog should be added somewhere to solve the undefined reference caused by protocolbuffers/protobuf#2719

A possible solution is that since the minimum cmake version is 3.13 now, we can set CMP0079 to NEW so that we can add dependency to protobuf-lite in onnxruntime's cmake.

snnn · 2019-04-02T03:13:18Z

In CMakeLists.txt, you may add "log" to onnxruntime_EXTERNAL_LIBRARIES.

snnn · 2019-04-02T03:14:10Z

Hi @manashgoswami, could you find someone verifying this PR? Thanks

daquexian · 2019-04-02T07:11:30Z

@snnn Thanks! I have updated my PR according to your advice. Now the tests compile.

snnn · 2019-04-02T07:41:31Z

Are you doing cross-compiling or native ?

daquexian · 2019-04-02T07:44:22Z

@snnn cross-compiling via android ndk.

snnn · 2019-04-02T07:50:41Z

the arch should be determined by the cmake official way

Any suggestion?

daquexian · 2019-04-02T08:23:39Z

@snnn My PR determines the architecture by CMAKE_ANDROID_ARCH_ABI, which is the variable used by CMake. This is what "cmake official way" means :)

Reference: The cmake manual for Android and CMAKE_ANDROID_ARCH_ABI.

yufenglee · 2019-04-02T23:47:07Z

cmake/CMakeLists.txt

@@ -581,6 +581,10 @@ else()
  set(onnxruntime_EXTERNAL_LIBRARIES_DEBUG ${onnxruntime_EXTERNAL_LIBRARIES})
 endif()

+if (CMAKE_SYSTEM_NAME STREQUAL "Android")
+  list(APPEND onnxruntime_EXTERNAL_LIBRARIES log)


What's this library for?

daquexian · 2019-04-04T05:25:14Z

Hi @snnn @yufenglee , is there still something I need to do for this PR? :)

pranavsharma · 2019-04-08T06:14:58Z

@daquexian can you please check-in a CI yaml file to ensure future changes don't break the Android build?

daquexian · 2019-04-12T04:04:53Z

@pranavsharma Hi, I have uploaded the new ci pipeline, however, I haven't tested it and I don't know how to enable it.

I build binary directly by cmake instead of build.py, so the python part is not needed. If you approve this way, I will remove all the code about python in android pipeline.

This reverts commit 3024632.

This reverts commit 1acfb67.

daquexian · 2019-04-17T14:14:21Z

I have tested this PR using my own azure account and verified the CI works(log: https://dev.azure.com/daquexian/onnxruntime/_build/results?buildId=14 for commit 1acfb67. I have reverted two commits which are only for my own azure pipeline). Now it can be tested in this Microsoft repo as long as an admin starts the CI. @linkerzhang @snnn @pranavsharma

And @linkerzhang , could I get the privilege to start the CI of this repo for the future development?

pranavsharma · 2019-04-17T17:12:08Z

/AzurePipelines run

azure-pipelines · 2019-04-17T17:12:55Z

Azure Pipelines successfully started running 7 pipeline(s), but failed to run 1 pipeline(s).

daquexian · 2019-04-17T23:15:20Z

@pranavsharma Thanks! Could you please enable the new Android CI for this PR? The yaml is tools/ci_build/github/azure-pipelines/android-arm64-crosscompile-ci-pipeline.yml

daquexian · 2019-04-18T02:02:45Z

@linkerzhang Thanks!

daquexian requested a review from a team as a code owner March 27, 2019 04:46

snnn mentioned this pull request Mar 27, 2019

deploy on ARM device(Android) #271

Closed

snnn reviewed Mar 27, 2019

View reviewed changes

cmake/onnxruntime_unittests.cmake Outdated Show resolved Hide resolved

snnn requested a review from a team March 27, 2019 08:55

raymondxyang assigned snnn Mar 29, 2019

daquexian force-pushed the android_build branch from a54537a to a8f04fe Compare March 30, 2019 12:27

daquexian added 3 commits April 2, 2019 15:03

enable android build

39fc0e4

Add 'log' to onnxruntime_EXTERNAL_LIBRARIES

d028c99

Remove cmake about header_files_test.cc

fdf552c

daquexian force-pushed the android_build branch from a8f04fe to fdf552c Compare April 2, 2019 07:04

yufenglee reviewed Apr 2, 2019

View reviewed changes

Add Android CI pipeline

e6f1567

daquexian closed this Apr 17, 2019

daquexian reopened this Apr 17, 2019

daquexian added 2 commits April 17, 2019 15:05

Remove some ms-specific(?) ci

3024632

Fix bash error

82d460e

daquexian added 2 commits April 17, 2019 15:33

Add execute flag for install_deps_android.sh

0fb7c49

Add install_ubuntu_for_android.sh

74e4e2e

daquexian force-pushed the android_build branch from 9a287e5 to 74e4e2e Compare April 17, 2019 08:01

daquexian added 7 commits April 17, 2019 16:10

Remove python in deps for android

faccc57

Add comment for BUILD_ARCH

c47f8e9

Set BUILD_SERVICE to cpu

1f1ecf5

Set BUILD_OS in run_build.sh

87a0dcc

Fix -o bug in run_build.sh

a2ec5ff

Android -> android

88f1c8d

Correct the android ndk location

a1c3e3a

daquexian force-pushed the android_build branch from 926aa19 to a1c3e3a Compare April 17, 2019 09:53

daquexian added 3 commits April 17, 2019 18:14

Checkout submodules in my own azure pipelines

1acfb67

Revert "Remove some ms-specific(?) ci"

40fb85f

This reverts commit 3024632.

Revert "Checkout submodules in my own azure pipelines"

e793bf1

This reverts commit 1acfb67.

linkerzhang approved these changes Apr 18, 2019

View reviewed changes

linkerzhang merged commit ac82c1f into microsoft:master Apr 18, 2019

daquexian deleted the android_build branch April 18, 2019 02:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable android build #715

enable android build #715

daquexian commented Mar 27, 2019 •

edited

msftclas commented Mar 27, 2019 •

edited

snnn commented Mar 27, 2019

daquexian commented Mar 27, 2019

daquexian commented Mar 30, 2019 •

edited

snnn commented Apr 2, 2019

snnn commented Apr 2, 2019

daquexian commented Apr 2, 2019 •

edited

snnn commented Apr 2, 2019

daquexian commented Apr 2, 2019

snnn commented Apr 2, 2019

daquexian commented Apr 2, 2019 •

edited

yufenglee Apr 2, 2019

daquexian commented Apr 4, 2019

pranavsharma commented Apr 8, 2019

daquexian commented Apr 12, 2019

daquexian commented Apr 17, 2019 •

edited

pranavsharma commented Apr 17, 2019

azure-pipelines bot commented Apr 17, 2019

daquexian commented Apr 17, 2019

daquexian commented Apr 18, 2019

enable android build #715

enable android build #715

Conversation

daquexian commented Mar 27, 2019 • edited

msftclas commented Mar 27, 2019 • edited

snnn commented Mar 27, 2019

daquexian commented Mar 27, 2019

daquexian commented Mar 30, 2019 • edited

snnn commented Apr 2, 2019

snnn commented Apr 2, 2019

daquexian commented Apr 2, 2019 • edited

snnn commented Apr 2, 2019

daquexian commented Apr 2, 2019

snnn commented Apr 2, 2019

daquexian commented Apr 2, 2019 • edited

yufenglee Apr 2, 2019

Choose a reason for hiding this comment

daquexian commented Apr 4, 2019

pranavsharma commented Apr 8, 2019

daquexian commented Apr 12, 2019

daquexian commented Apr 17, 2019 • edited

pranavsharma commented Apr 17, 2019

azure-pipelines bot commented Apr 17, 2019

daquexian commented Apr 17, 2019

daquexian commented Apr 18, 2019

daquexian commented Mar 27, 2019 •

edited

msftclas commented Mar 27, 2019 •

edited

daquexian commented Mar 30, 2019 •

edited

daquexian commented Apr 2, 2019 •

edited

daquexian commented Apr 2, 2019 •

edited

daquexian commented Apr 17, 2019 •

edited