Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SoLoader can't find .so deps when app is packaged as Prebuilt APK with Android AOSP Source #28

Closed
hhff opened this issue Feb 8, 2019 · 10 comments

Comments

@hhff
Copy link

hhff commented Feb 8, 2019

Thanks so much for the hard work here!

I have a unique situation that perhaps you can shed some light on!

I am building a React Native Android application that will ship with an Android Device, and therefore it's being included in the AOSP build process as a prebuilt APK.

When I install the APK via ADB, it runs great! However, when I build the Android source including the App, I flash the device, and see the App on the homescreen. However, when it boots, I see the following logs from SoLoader:

01-08 06:20:53.366 2950-2997/com.lightos D/SoLoader: Loading lib dependencies: [libglog_init.so, libyoga.so, libprivatedata.so, libfb.so, libfolly_json.so, libjsc.so, libglog.so, libgnustl_shared.so, libandroid.so, liblog.so, libstdc++.so, libm.so, libc.so, libdl.so]
    About to load: libglog_init.so
01-08 06:20:53.368 695-717/? D/ActivityTrigger: ActivityTrigger activityStopTrigger 
01-08 06:20:53.373 3430-3514/? W/Adreno-ES20: <core_glFinish:34>: glFinish skipped: 0
01-08 06:20:53.376 3430-3514/? W/Adreno-ES20: <core_glFinish:34>: glFinish skipped: 0
01-08 06:20:53.465 2950-2997/com.lightos D/SoLoader: Loading lib dependencies: [libglog.so, libgnustl_shared.so, liblog.so, libstdc++.so, libm.so, libc.so, libdl.so]
    About to load: libglog.so
01-08 06:20:53.515 2950-2997/com.lightos D/SoLoader: Loading lib dependencies: [libgnustl_shared.so, libstdc++.so, libm.so, libc.so, libdl.so]
01-08 06:20:53.516 2950-2997/com.lightos D/SoLoader: About to load: libgnustl_shared.so
01-08 06:20:53.550 2950-2997/com.lightos D/SoLoader: Loading lib dependencies: [libm.so, libc.so, libdl.so]
    About to load: libm.so
01-08 06:20:53.551 2950-2997/com.lightos D/SoLoader: libm.so not found on /data/data/com.lightos/lib-main
    Result 0 for libm.so in source com.facebook.soloader.ApkSoSource[root = /data/data/com.lightos/lib-main flags = 1]
    Extraction logs: null
01-08 06:20:53.552 2950-2997/com.lightos D/SoLoader: libm.so not found on /vendor/lib
    Result 0 for libm.so in source com.facebook.soloader.DirectorySoSource[root = /vendor/lib flags = 2]
01-08 06:20:53.553 2950-2997/com.lightos D/SoLoader: Loaded: libm.so
01-08 06:20:53.554 2950-2997/com.lightos D/SoLoader: About to load: libc.so
    libc.so not found on /data/data/com.lightos/lib-main
01-08 06:20:53.555 2950-2997/com.lightos D/SoLoader: Result 0 for libc.so in source com.facebook.soloader.ApkSoSource[root = /data/data/com.lightos/lib-main flags = 1]
01-08 06:20:53.556 2950-2997/com.lightos D/SoLoader: Extraction logs: null
    libc.so not found on /vendor/lib
01-08 06:20:53.557 2950-2997/com.lightos D/SoLoader: Result 0 for libc.so in source com.facebook.soloader.DirectorySoSource[root = /vendor/lib flags = 2]
    Loaded: libc.so
01-08 06:20:53.558 2950-2997/com.lightos D/SoLoader: About to load: libdl.so
    libdl.so not found on /data/data/com.lightos/lib-main
01-08 06:20:53.559 2950-2997/com.lightos D/SoLoader: Result 0 for libdl.so in source com.facebook.soloader.ApkSoSource[root = /data/data/com.lightos/lib-main flags = 1]
    Extraction logs: null
01-08 06:20:53.560 2950-2997/com.lightos D/SoLoader: libdl.so not found on /vendor/lib
    Result 0 for libdl.so in source com.facebook.soloader.DirectorySoSource[root = /vendor/lib flags = 2]
    Loaded: libdl.so
01-08 06:20:53.563 2950-2997/com.lightos E/SoLoader: Error when loading lib: dlopen failed: couldn't map "/data/data/com.lightos/lib-main/libgnustl_shared.so" segment 2: Permission denied
01-08 06:20:53.564 2950-2997/com.lightos E/SoLoader: Could not load: libgnustl_shared.so
    Could not load: libglog.so
    Could not load: libglog_init.so
01-08 06:20:53.565 2950-2997/com.lightos E/SoLoader: Could not load: libreactnativejni.so
01-08 06:20:53.570 2950-2997/com.lightos E/AndroidRuntime: FATAL EXCEPTION: Thread-2
    Process: com.lightos, PID: 2950
    java.lang.UnsatisfiedLinkError: couldn't find DSO to load: libreactnativejni.so
        at com.facebook.soloader.SoLoader.doLoadLibraryBySoName(SoLoader.java:522)
        at com.facebook.soloader.SoLoader.loadLibraryBySoName(SoLoader.java:420)
        at com.facebook.soloader.SoLoader.loadLibrary(SoLoader.java:370)
        at com.facebook.soloader.SoLoader.loadLibrary(SoLoader.java:335)
        at com.facebook.react.bridge.ReactBridge.staticInit(ReactBridge.java:18)
        at com.facebook.react.bridge.NativeMap.<clinit>(NativeMap.java:19)
        at com.facebook.react.bridge.JSCJavaScriptExecutorFactory.create(JSCJavaScriptExecutorFactory.java:21)
        at com.facebook.react.ReactInstanceManager$5.run(ReactInstanceManager.java:917)
        at java.lang.Thread.run(Thread.java:764)
01-08 06:20:53.547 2950-2950/com.lightos W/Thread-2: type=1400 audit(0.0:74): avc: denied { execute } for path="/data/data/com.lightos/lib-main/libgnustl_shared.so" dev="mmcblk0p38" ino=2430 scontext=u:r:system_app:s0 tcontext=u:object_r:system_app_data_file:s0 tclass=file permissive=0

It appears to me that the install process that moves the .so files to the appropriate place on the Android filesystem isn't being run for the packaged app in the AOSP build step (but I have confirmed they are in the APK itself by unzipping and checking).

What should I do to get this working as an Android prebuilt?

Thank you!

@hhff
Copy link
Author

hhff commented Feb 8, 2019

Oh - and some extra context:

I've tried unpacking all of the .so files from the APK, and adding them as prebuilts to the AOSP build process, like so (via the Android.mk file):

LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)
LOCAL_MODULE := libreactnativejni
LOCAL_SRC_FILES := libreactnativejni.so
LOCAL_MODULE_SUFFIX := .so
LOCAL_MODULE_CLASS := SHARED_LIBRARIES
LOCAL_MODULE_TAGS := optional
include $(BUILD_PREBUILT)

and adding

LOCAL_REQUIRED_MODULES := libfb libfolly_json libglog_init libglog libgnustl_shared libicu_common libimagepipeline libjsc libprivatedata libreactnativejni libyoga

to the prebuilt block itself

but to no avail.

@passy
Copy link
Member

passy commented Feb 11, 2019

Sadly, this is way out of my comfort zone, so I can't give you any good advice here but the main issue appears to be

Error when loading lib: dlopen failed: couldn't map "/data/data/com.lightos/lib-main/libgnustl_shared.so" segment 2: Permission denied

I'm not sure if there are certain filesystem/mount flags that would prevent mapping the file from that location or if it's a security restriction in the Android runtime only allowing dlopen from application-specific paths.

Googling for "dlopen failed: couldn't map" brings up a lot of Android-related issues, but nothing particularly actionable. Others have suggested copying the files to an application-private location. Perhaps you could do that as a workaround on startup.

@hhff
Copy link
Author

hhff commented Feb 11, 2019

Cool yeah. I have a feeling that "permission denied" is actually because the file doesn't exist. I'm still working on this, but when I solve it I'll come back and update this issue to help anyone else who comes this way.

@SergiiGudym
Copy link

Hi. Did you fix it?

@hhff
Copy link
Author

hhff commented Mar 18, 2019

hi @SergiiGudym - yup I did. So - for anyone else:

If you're embedding your APK into a AOSP build that will packaged with the ROM, it's important to note a few things:

  • the APK isn't going to go through an "Install" process (like it would via ADB or any other traditional install process). As part of that normal install process, packaged, prebuilt libs will be extracted and moved to the appropriate place for the APK to require them. This doesn't happen when packaging the APK with AOSP.

  • an APK is basically a fancy zip file, and those static libs are inside.


So - when you're packaging your React Native app with an AOSP ROM, you'll need to do the following:

  1. Do all of the normal setup you would with a standard APK that you're building with AOSP (some googling will help with this)

  2. Unzip your APK:

mkdir ./unzipped && unzip ./MyAPK.apk -d ./unzipped
  1. Move your static libs to the same level as your Android.mk (assuming you're building for both architectures as per react native default)
mv ./unzipped/lib/armeabi-v7a ./lib
mv ./unzipped/lib/x86 ./lib64
  1. Edit your Android.mk file to move these libs during AOSP build time:
LOCAL_PATH := $(call my-dir)

# Example for 32 bit deps
include $(CLEAR_VARS)
LOCAL_MODULE := libfb
LOCAL_SRC_FILES := lib/libfb.so
LOCAL_MODULE_TAGS := optional
LOCAL_MULTILIB := 32
LOCAL_MODULE_SUFFIX := .so
LOCAL_MODULE_CLASS := SHARED_LIBRARIES
LOCAL_MODULE_PATH := $(PRODUCT_OUT)/vendor/lib
include $(BUILD_PREBUILT)

# Example for 64 bit deps
include $(CLEAR_VARS)
LOCAL_MODULE := libfb
LOCAL_SRC_FILES := lib64/libfb.so
LOCAL_MODULE_TAGS := optional
LOCAL_MULTILIB := 64
LOCAL_MODULE_SUFFIX := .so
LOCAL_MODULE_CLASS := SHARED_LIBRARIES
LOCAL_MODULE_PATH := $(PRODUCT_OUT)/vendor/lib64
include $(BUILD_PREBUILT)

# ^^At the time of writing, React Native has like ~11 static libs, so you'll have ~22 of these blocks

# Finally, declare the APK itself
include $(CLEAR_VARS)
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE := MyAPK
LOCAL_SRC_FILES := $(LOCAL_MODULE).apk
LOCAL_MODULE_CLASS := APPS
LOCAL_MODULE_SUFFIX := $(COMMON_ANDROID_PACKAGE_SUFFIX)
LOCAL_CERTIFICATE := platform
LOCAL_PRIVILEGED_MODULE := true
include $(BUILD_PREBUILT)
  1. Now when you build AOSP, flash the device, and run MyAPK, SoLoader should be able to find and link these libs at runtime.

Some gotchas:

  • The Android.mk file is super finicky. Make sure you don't have any extra spaces at the end of lines, etc. Seriously! It can crash an AOSP build.
  • Remember your static libs may change if your build does (Upgrading react native, etc). We've setup a shell script to pull in the latest APK from an s3 bucket, and automate most of the above. Then when we're about to check in the new version of the APK, we'll see any libs were added. If so, we'll add a block for them in the Android.mk before committing.
  • Hit me with any more questions!

@hhff
Copy link
Author

hhff commented Mar 18, 2019

@passy this can be closed, fwiw. Thought i'd leave it open in case anyone has more questions!

@ozymand1as
Copy link

@hhff Is there anything else that has to be done? I've added everything to Android.mk, but my app still doesn't run and the libraries don't even appear in the target folder.

@ozymand1as
Copy link

ozymand1as commented Apr 25, 2019

It seems that it's also necessary to add all the libraries to your app's module in Android.mk with LOCAL_SHARED_LIBRARIES.
So, in my Android.mk app is declared like this:

#all .so libraries declared there ^^^

include $(CLEAR_VARS)
LOCAL_MODULE_TAGS := optional
LOCAL_MODULE := <Module name>
LOCAL_CERTIFICATE := platform
LOCAL_SRC_FILES := <apk file>
LOCAL_MODULE_CLASS := APPS
LOCAL_PRIVILEGED_MODULE := true
LOCAL_MODULE_SUFFIX := $(COMMON_ANDROID_PACKAGE_SUFFIX)
LOCAL_SHARED_LIBRARIES := libfb libfolly_json libglog_init libglog libgnustl_shared libicu_common libimagepipeline libjsc libprivatedata libreactnativejni libyoga
include $(BUILD_PREBUILT)

@hhff
Copy link
Author

hhff commented Apr 25, 2019

Cool thanks @ozymand1as ! FWIW - I didn't have to do that, but it's likely we're running different versions of AOSP so perhaps thats the reason

facebook-github-bot pushed a commit that referenced this issue May 1, 2019
Summary:
Fix to allow SoLoader to find .so deps when app is packaged as Prebuilt APK with Android AOSP Source
When an apk has .so files as part of the APK, the Android make system will generate the APK so as to allow .so files to be loaded directly.

Another option is to include the .so in system libs by updating the Android make file but that means that the files are shared across the OS and can cause conflicts with
applications that may want their own version of the .so

Possible fix for issue
#28

How to use:
  SoLoader.init(mContext, SOLOADER_DISABLE_BACKUP_SOSOURCE);
  SoLoader.setSystemLoadLibraryWrapper(
        new SystemLoadLibraryWrapper() {
          Override
          public void loadLibrary(String libName) {
            System.loadLibrary(libName);
          }
        });

Reviewed By: joelmccall

Differential Revision: D15082630

fbshipit-source-id: 68db389bc105929c6cba7a92285c5a3df947f51c
@simpleton
Copy link
Member

Version 0.10.3 has been released which includes this fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants