Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not build android_library with embedded so file (using ndk) #3924

Closed
spinorx opened this issue Oct 19, 2017 · 13 comments
Closed

Can not build android_library with embedded so file (using ndk) #3924

spinorx opened this issue Oct 19, 2017 · 13 comments
Assignees
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) team-Android Issues for Android team type: bug

Comments

@spinorx
Copy link

spinorx commented Oct 19, 2017

If possible, provide a minimal example to reproduce the problem:

Created a simple android_library which encloses java and needed so file at: https://github.com/spinorx/bazel_android_test

doing: bazel build --config android :and_lib -s

Getting compilation error:

ERROR: /Users/param/bazel_android_test/BUILD:3:1: C++ compilation of rule '//:liband_cpp.so' failed (Exit 1).
clang: error: unknown argument: '-m<platform_for_version_min>-version-min=11.0'
Target //:and_lib failed to build

I am trying to get android ndk to build so file using libc++ using bazel.
(Note that I has to push include under libcxx for libc++ because of:
#3923)

Environment info

  • Operating System:
    Mac OS 10.12.6

  • Bazel version (output of bazel info release):
    release 0.6.1-homebrew

Have you found anything relevant by searching the web?

(e.g. StackOverflow answers,
GitHub issues,
email threads on the bazel-discuss Google group)

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

@spinorx
Copy link
Author

spinorx commented Oct 19, 2017

command issued is also using local clang rather than one in android ndk.

[param@param-mac:~/bazel_android_test]$ bazel build --config android :and_lib -s
INFO: Found 1 target...
>>>>> # //:liband_cpp.so [action 'Compiling jni.cc']
(cd /private/var/tmp/_bazel_param/cf03584cc2c459d7c3898fe4f603bb7c/execroot/__main__ && \
  exec env - \
    APPLE_SDK_PLATFORM='' \
    APPLE_SDK_VERSION_OVERRIDE='' \
    PATH=.:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/param/Library/Android/sdk/platform-tools:/Users/param/android-ndk-r14b:/Developer/NVIDIA/CUDA-8.0/bin \
    TMPDIR=/var/folders/6v/kqb7kchj0kg5wgxqjh3bp73m0000gn/T/ \
    XCODE_VERSION_OVERRIDE=9.0.1 \
  external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -O0 -DDEBUG '-std=c++11' '-std=c++14' -iquote . -iquote bazel-out/stub_armeabi-v7a-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/stub_armeabi-v7a-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -MD -MF bazel-out/stub_armeabi-v7a-fastbuild/bin/_objs/liband_cpp.so/jni.pic.d '-frandom-seed=bazel-out/stub_armeabi-v7a-fastbuild/bin/_objs/liband_cpp.so/jni.pic.o' -fPIC '-isysroot __BAZEL_XCODE_SDKROOT__' '-m<platform_for_version_min>-version-min=11.0' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c jni.cc -o bazel-out/stub_armeabi-v7a-fastbuild/bin/_objs/liband_cpp.so/jni.pic.o)
>>>>> # //:and_lib [action 'Validating Android resources for //:and_lib']
(cd /private/var/tmp/_bazel_param/cf03584cc2c459d7c3898fe4f603bb7c/execroot/__main__ && \
  exec env - \
    PATH=.:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/param/Library/Android/sdk/platform-tools:/Users/param/android-ndk-r14b:/Developer/NVIDIA/CUDA-8.0/bin \
    TMPDIR=/var/folders/6v/kqb7kchj0kg5wgxqjh3bp73m0000gn/T/ \
  bazel-out/host/bin/external/bazel_tools/src/tools/android/java/com/google/devtools/build/android/ResourceProcessorBusyBox --tool VALIDATE -- --buildToolsVersion 26.0.2 --aapt bazel-out/host/bin/external/android_sdk/aapt_binary --annotationJar external/android_sdk/tools/support/annotations.jar --androidJar external/android_sdk/platforms/android-25/android.jar --mergedResources bazel-out/stub_armeabi-v7a-fastbuild/bin/and_lib_files/resource_files.zip --manifest bazel-out/stub_armeabi-v7a-fastbuild/bin/and_lib_processed_manifest/AndroidManifest.xml --debug --rOutput bazel-out/stub_armeabi-v7a-fastbuild/bin/and_lib_symbols/R.txt --srcJarOutput bazel-out/stub_armeabi-v7a-fastbuild/bin/and_lib.srcjar --packagePath bazel-out/stub_armeabi-v7a-fastbuild/bin/and_lib_files/library.ap_)
ERROR: /Users/param/bazel_android_test/BUILD:3:1: C++ compilation of rule '//:liband_cpp.so' failed (Exit 1).
clang: error: unknown argument: '-m<platform_for_version_min>-version-min=11.0'
Target //:and_lib failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.693s, Critical Path: 0.49s
[param@param-mac:~/bazel_android_test]$

When I add android_binary it seems to cross that but gets stuck at some linking error (-lc++_static not found, even though libc++_static.a is referred directly too):

[param@param-mac:~/bazel_android_test]$ bazel build --config android :and_bin -s
INFO: Found 1 target...
>>>>> # //:liband_cpp.so [action 'Linking liband_cpp.so']
(cd /private/var/tmp/_bazel_param/cf03584cc2c459d7c3898fe4f603bb7c/execroot/__main__ && \
  exec env - \
    PATH=.:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/param/Library/Android/sdk/platform-tools:/Users/param/android-ndk-r14b:/Developer/NVIDIA/CUDA-8.0/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/var/folders/6v/kqb7kchj0kg5wgxqjh3bp73m0000gn/T/ \
  external/android_ndk/ndk/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang -shared -o bazel-out/android-arm-linux-androideabi-clang3.8-v7a-libcpp-fastbuild/bin/liband_cpp.so -Wl,-whole-archive bazel-out/android-arm-linux-androideabi-clang3.8-v7a-libcpp-fastbuild/bin/_objs/liband_cpp.so/jni.o -Wl,-no-whole-archive external/android_ndk/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a/libandroid_support.a external/android_ndk/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a/libc++.a external/android_ndk/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a/libc++_static.a external/android_ndk/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a/libc++abi.a external/android_ndk/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a/libunwind.a -llog -static-libgcc -gcc-toolchain external/android_ndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64 -no-canonical-prefixes -target armv7-none-linux-androideabi -Wl,--fix-cortex-a8 -Wl,-S '--sysroot=external/android_ndk/ndk/platforms/android-24/arch-arm')
>>>>> # //:and_bin [action 'Processing Android resources for //:and_bin']
(cd /private/var/tmp/_bazel_param/cf03584cc2c459d7c3898fe4f603bb7c/execroot/__main__ && \
  exec env - \
    PATH=.:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/param/Library/Android/sdk/platform-tools:/Users/param/android-ndk-r14b:/Developer/NVIDIA/CUDA-8.0/bin \
    TMPDIR=/var/folders/6v/kqb7kchj0kg5wgxqjh3bp73m0000gn/T/ \
  bazel-out/host/bin/external/bazel_tools/src/tools/android/java/com/google/devtools/build/android/ResourceProcessorBusyBox --tool PACKAGE -- --aapt bazel-out/host/bin/external/android_sdk/aapt_binary --primaryData ::AndroidManifest.xml --buildToolsVersion 26.0.2 --annotationJar external/android_sdk/tools/support/annotations.jar --androidJar external/android_sdk/platforms/android-25/android.jar --rOutput bazel-out/stub_armeabi-v7a-fastbuild/bin/and_bin_symbols/R.txt --srcJarOutput bazel-out/stub_armeabi-v7a-fastbuild/bin/and_bin.srcjar --proguardOutput bazel-out/stub_armeabi-v7a-fastbuild/bin/proguard/and_bin/_and_bin_proguard.cfg --mainDexProguardOutput bazel-out/stub_armeabi-v7a-fastbuild/bin/proguard/and_bin/main_dex_and_bin_proguard.cfg --manifestOutput bazel-out/stub_armeabi-v7a-fastbuild/bin/and_bin_processed_manifest/AndroidManifest.xml --resourcesOutput bazel-out/stub_armeabi-v7a-fastbuild/bin/and_bin_files/resource_files.zip --packagePath bazel-out/stub_armeabi-v7a-fastbuild/bin/and_bin.ap_ --debug)
>>>>> # @android_sdk//com.android.support:support-annotations-25.3.1 [action 'Dexing external/android_sdk/com.android.support/_dx/support-annotations-25.3.1/support-annotations-25.3.1.jar_desugared.jar with applicable dexopts []']
(cd /private/var/tmp/_bazel_param/cf03584cc2c459d7c3898fe4f603bb7c/execroot/__main__ && \
  exec env - \
    PATH=.:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/param/Library/Android/sdk/platform-tools:/Users/param/android-ndk-r14b:/Developer/NVIDIA/CUDA-8.0/bin \
    TMPDIR=/var/folders/6v/kqb7kchj0kg5wgxqjh3bp73m0000gn/T/ \
  bazel-out/host/bin/external/bazel_tools/src/tools/android/java/com/google/devtools/build/android/dexer/DexBuilder @bazel-out/android-arm-linux-androideabi-clang3.8-v7a-libcpp-fastbuild/bin/external/android_sdk/com.android.support/_dx/support-annotations-25.3.1/support-annotations-25.3.1.jar.dex.zip-2.params)
ERROR: /Users/param/bazel_android_test/BUILD:3:1: Linking of rule '//:liband_cpp.so' failed (Exit 1).
external/android_ndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lc++_static
external/android_ndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lc++abi
external/android_ndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -landroid_support
external/android_ndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld: error: cannot find -lunwind
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:and_bin failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.950s, Critical Path: 0.77s
[param@param-mac:~/bazel_android_test]$

@spinorx
Copy link
Author

spinorx commented Oct 19, 2017

@aj-michael who is doing most android changes.

@aj-michael
Copy link
Contributor

  • @ahumesky , because I'm not sure I'll have time to look at this week.

What is your NDK version?

This is almost certainly the same bug that was reported in #3641. I started a fix a bit ago, but have not submitted it.

@spinorx
Copy link
Author

spinorx commented Oct 19, 2017

Thanks for response.

I am using android-ndk-r14b/

I don't think this is include path related. Even the clang tool which is used seems to be wrong when going through android_library (using external/local_config_cc/wrapped_clang rather than external/android_ndk/ndk/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang). Note in bazelrc I am using --android_crosstool_top=@android_ndk//:toolchain-libcpp rather than blanket --crosstool_top=//external:android/crosstool.

If I build through android_binary rather than android_library things work well. Also need to explicitly add lib path --linkopt=-Lexternal/android_ndk/ndk/sources/cxx-stl/llvm-libc++/libs/armeabi-v7a to build. (This is on top of include file workaround mentioned in #3923)

@ahumesky
Copy link
Contributor

ahumesky commented Oct 20, 2017

The more typical way for an android app to depend on native code is to depend on a cc_library directly:

android_binary(
  name = "and_bin",
  srcs = glob([
    "java/**/*.java",
  ]),
  custom_package = "",
  manifest = "AndroidManifest.xml",
  deps = [
    ":native_deps",
    "@android_sdk//com.android.support:support-annotations-25.3.1",
  ],
)

cc_library(
  name = "native_deps",
  srcs = [
    "jni.cc",
  ],
)

The reason that local clang was being used was that you were building the android_library as the top-level target, and not the android_binary. It's unfortunately a bit complicated. android_binary has special treatment for its native deps (cc_library rules) where they will be built in the "android configuration", which is controlled by --fat_apk_cpu and --android_crosstool_top (among others). --android_crosstool_top is automatically set to the NDK if you use an android_ndk_repository rule, and --fat_apk_cpu defaults to armeabi-v7a. It can be set to armeabi-v7a,x86 to create two .so files in the apk, one for each architecture. Since the change to the android configuration only happens for android_binary, when you were building the android_library directly, the CC rules below it didn't get the android configuration, and hence were getting the "normal" CC configuration (whatever --cpu and --crosstool_top were set to).

Basically, all you should need to do is use a cc_library as in the example above and invoke bazel build :and_bin (no --config android required). If you want to build the native code for an x86 device (e.g. an emulator) then you can add --fat_apk_cpu=x86.

If you still want to be able to build the android_library as a top-level target (i.e. from the command line directly), then you'll need to set the CC configuration to android: --crosstool_top=@android_ndk//:toolchain-libcpp --cpu=armeabi-v7a --host_crosstool_top=@bazel_tools//tools/cpp:toolchain. Note that this changes the CC configuration for the whole build to android (not just the dependencies of android_binary rules), so if you wanted to build, say, some other cc_binary in the same invocation, or your entire codebase (e.g. //...) which might include non-android things, then they probably won't build.

See https://docs.bazel.build/versions/master/user-manual.html#flag--fat_apk_cpu

@spinorx
Copy link
Author

spinorx commented Oct 20, 2017

Thanks Adam. I was expecting android_library to behave like android_binary in terms of changing cc_library's special treatment since what one can deduce from android_binary can be deduced from android library. My initial plan was to use bagel to produce the aar file which included so file (the model pieces) and use android studio project for UI pieces. Last paragraph will help with that.

FWIW iOS_framework behaves similar to ios_application in terms of changing the cc_library to use ios_* settings.

Given workarounds definitely sounds like lower priority. (Assuming the linking issues with libc++ will be addressed with the include issue).

@spinorx
Copy link
Author

spinorx commented Oct 21, 2017

Hi Adam, Another question partially related to cross tool top. What is good way to detect bazel is building android. In protobuf lib I see:
config_setting(
name = "android",
values = {
"crosstool_top": "//external:android/crosstool",
},

Which does not look compatible with using --android_crosstool_top.
I added flag which can be passed in with --define PLATFORM=android. But not sure if there is better approach.

protocolbuffers/protobuf#3778

@aj-michael
Copy link
Contributor

That's a hard question, because Bazel doesn't really know about "building android". When bazel analyzes cc_library rules, it uses the values of --crosstool_top and --cpu. When a Bazel analyzes an android_binary, it changes the values of --crosstool_top and --cpu based on --android_crosstool_top, --android_cpu and --fat_apk_cpu. Furthermore, while the Bazel team provides some built-in crosstools like //external:android/crosstool, there is nothing to stop a third_party from creating their own crosstool with a different name.

Protobuf's config_setting looks like it should support your use case, if you are building an android_binary with --android_crosstool_top. Because the cc_librarys underneath it will see --crosstool_top set to --android_crosstool_top.

@spinorx
Copy link
Author

spinorx commented Oct 23, 2017

It was not working because I am using --android_crosstool_top=@android_ndk//:toolchain-libcpp. I have to use that because grpc uses some cpp functions which are not in libstdcpp: https://stackoverflow.com/questions/22774009/android-ndk-stdto-string-support

Adding:
config_setting(
name = "android_libcpp",
values = {
"crosstool_top": "@android_ndk//:toolchain-libcpp",
},
)

works.

@xfxyjwf - Does adding this to protobuf BUILD sound reasonable?

@aj-michael
Copy link
Contributor

aj-michael commented Oct 23, 2017

So just to summarize, the issue here is that "building for Android" can mean multiple things and it's not reasonable for protobuf team to keep an exhaustive list of all of them. (I'd argue it's also not reasonable for Bazel team to keep an exhaustive list, since users can define their own Android crosstools. But maybe that's another discussion.)

@spinorx , you can try adding this to the bottom of your WORKSPACE:

bind(name = "android/crosstool", actual = "@android_ndk//:toolchain-libcpp")

This will override the value of //external:android/crosstool from @android_ndk//:gnu-libstdcpp to @android_ndk//:toolchain-libcpp. Then you can use --android_crosstool_top=//external:android/crosstool.

This solution is essentially "make the user specify that their crosstool is an android crosstool".

@spinorx
Copy link
Author

spinorx commented Oct 23, 2017

Thanks Adam, this is much more elegant solution.

@aj-michael
Copy link
Contributor

FWIW, we have a long-term solution called "constraint-based platforms" that is still in the works. It should us to express things like, "this cross tool is usable for building against the Android platform". And the config settings can select on the well-defined "Android platform". You can read a bit about it here:

https://docs.google.com/document/d/1-G-VPRLEj9VyfC6VrQBiR8To-dZjnBSQS66Y4nargGM

@jin jin added team-Android Issues for Android team and removed category: rules > android labels Aug 11, 2018
@jin jin added the P3 We're not considering working on this, but happy to review a PR. (No assignee) label Feb 16, 2019
@jin
Copy link
Member

jin commented Oct 24, 2019

Closing as Adam has provided a solution, and we are moving towards a new configurability solution.

@jin jin closed this as completed Oct 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 We're not considering working on this, but happy to review a PR. (No assignee) team-Android Issues for Android team type: bug
Projects
None yet
Development

No branches or pull requests

4 participants