New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support DSOs embedded in the .apk #2154

Merged
merged 2 commits into from Sep 17, 2018

Conversation

Projects
None yet
3 participants
@grendello
Contributor

grendello commented Sep 6, 2018

Context: #1906

Support DSOs embedded in the APK

Android API 23 introduced a new way of dealing with the native shared libraries
shipped in the APK. Before that API level, the libraries would be always
extraced and placed in the application data directory, thus occupying more space
than necessary. API 23 added a new manifest <application> element attribute,
android:extractNativeLibs, which if set makes Android not extract the
libraries to the filesystem. API 23 added a way to load those libraries directly
from the APK. In order to support that there are a few requirements which this
commit implements:

  • DSO (.so) files must be stored uncompressed in the APK
  • <application android:extractNativeLibs="false"/> must be present
  • DSOs in the APK must be aligned on the memory page boundary (the -p flag
    passed to zipalign takes care of that

This commit also implements libmonodroid suport for loading our DSOs directly
from the APK. This operation mode is enabled by the presence of the
__XA_DSO_IN_APK environment variable. This variable is inserted into the
application's environment by way of placing it in the environment file (a file
part of the XA project that is marked with the AndroidEnvironment build
action). In that mode, the DSOs are no longer looked up in the application
data directory but only in the override directories (if the APK is built in
Debug configuration) and in the APK itself.

Currently, in order to activate the above mode, one has to perform the following
actions manually:

  • Add the android:extractNativeLibs="false" attribute to the <application>
    element in the Properties/AndroidManifest.xml file

  • Add the following property to the project file:

      <AndroidStoreUncompressedFileExtensions>.so</AndroidStoreUncompressedFileExtensions>
  • Add an android environment file to the project with a line which says
      __XA_DSO_IN_APK=1

After that the application should work in the embedded DSO mode without
problems.

A couple of tests are provided to test building and execution of embedded DSO
application on device, as well as to validate the built APK.

@grendello grendello requested a review from jonpryor Sep 6, 2018

@grendello

This comment has been minimized.

Show comment
Hide comment
@grendello

grendello Sep 7, 2018

Contributor

Hmm, and the build failed because of

04:42:45  > git clean -fdx # timeout=10
04:43:11 FATAL: Command "git clean -fdx" returned status code 1:
04:43:11 Removing xamarin.android-oss_v9.1.99.161_Darwin-x86_64_HEAD_a9f0db2/
04:43:11 
04:43:11 stderr: warning: failed to remove bin/TestRelease/temp: Directory not empty
04:43:11 
04:43:11 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 167.220.148.18/167.220.148.18:44113
Contributor

grendello commented Sep 7, 2018

Hmm, and the build failed because of

04:42:45  > git clean -fdx # timeout=10
04:43:11 FATAL: Command "git clean -fdx" returned status code 1:
04:43:11 Removing xamarin.android-oss_v9.1.99.161_Darwin-x86_64_HEAD_a9f0db2/
04:43:11 
04:43:11 stderr: warning: failed to remove bin/TestRelease/temp: Directory not empty
04:43:11 
04:43:11 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 167.220.148.18/167.220.148.18:44113

@grendello grendello requested a review from dellis1972 as a code owner Sep 7, 2018

@grendello

This comment has been minimized.

Show comment
Hide comment
@grendello

grendello Sep 13, 2018

Contributor

The failure is because of a mono test crash that wasn't caught by NUnit:

09-12 20:24:06.297  4797  4815 I NUnit   : Read  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    : Unhandled Exception: (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    : System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Net.Sockets.SocketFlags socketFlags) [0x00017] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referen
cesource/System/net/System/Net/Sockets/Socket.cs:1773  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x0009b] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/System/net/System/Net/Sockets/N
etworkStream.cs:513  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :    --- End of inner exception stack trace --- (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x000c3] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/System/net/System/Net/Sockets/N
etworkStream.cs:525  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Net.RequestStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00033] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/System/System.Net/RequestStream.cs:141  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at MonoTests.System.Net.StreamExtensions.ReadAll (System.IO.Stream stream, System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00004] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at MonoTests.System.Net.HttpWebRequestTest+<>c__DisplayClass63_0.<PrematureStreamCloseAborts>b__1 (System.Net.HttpListenerContext c) [0x0000b] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at MonoTests.System.Net.HttpWebRequestTest+ListenerScope+<>c__DisplayClass4_0.<RequestHandler>b__0 (System.Object o) [0x00000] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x0000d] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1308  
(TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in /Users/builder/jenkins/workspace/xamarin-a
ndroid/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:961  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /Users/builder/jenkins/workspace/xamarin-android/x
amarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:908  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadp
ool.cs:1285  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:858  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1213  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt : [ERROR] FATAL UNHANDLED EXCEPTION: System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Net.Sockets.SocketFlags socketFlags) [0x00017] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referen
cesource/System/net/System/Net/Sockets/Socket.cs:1773  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x0009b] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/System/net/System/Net/Sockets/N
etworkStream.cs:513  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :    --- End of inner exception stack trace --- (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x000c3] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/System/net/System/Net/Sockets/N
etworkStream.cs:525  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Net.RequestStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00033] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/System/System.Net/RequestStream.cs:141  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at MonoTests.System.Net.StreamExtensions.ReadAll (System.IO.Stream stream, System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00004] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at MonoTests.System.Net.HttpWebRequestTest+<>c__DisplayClass63_0.<PrematureStreamCloseAborts>b__1 (System.Net.HttpListenerContext c) [0x0000b] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at MonoTests.System.Net.HttpWebRequestTest+ListenerScope+<>c__DisplayClass4_0.<RequestHandler>b__0 (System.Object o) [0x00000] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x0000d] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1308  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:961  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:908  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1285  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:858  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1213  (TaskId:131)
Contributor

grendello commented Sep 13, 2018

The failure is because of a mono test crash that wasn't caught by NUnit:

09-12 20:24:06.297  4797  4815 I NUnit   : Read  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    : Unhandled Exception: (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    : System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Net.Sockets.SocketFlags socketFlags) [0x00017] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referen
cesource/System/net/System/Net/Sockets/Socket.cs:1773  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x0009b] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/System/net/System/Net/Sockets/N
etworkStream.cs:513  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :    --- End of inner exception stack trace --- (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x000c3] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/System/net/System/Net/Sockets/N
etworkStream.cs:525  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Net.RequestStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00033] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/System/System.Net/RequestStream.cs:141  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at MonoTests.System.Net.StreamExtensions.ReadAll (System.IO.Stream stream, System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00004] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at MonoTests.System.Net.HttpWebRequestTest+<>c__DisplayClass63_0.<PrematureStreamCloseAborts>b__1 (System.Net.HttpListenerContext c) [0x0000b] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at MonoTests.System.Net.HttpWebRequestTest+ListenerScope+<>c__DisplayClass4_0.<RequestHandler>b__0 (System.Object o) [0x00000] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x0000d] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1308  
(TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in /Users/builder/jenkins/workspace/xamarin-a
ndroid/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:961  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /Users/builder/jenkins/workspace/xamarin-android/x
amarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:908  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadp
ool.cs:1285  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:858  (TaskId:131)
  09-12 20:24:06.297  4797  7962 E mono    :   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1213  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt : [ERROR] FATAL UNHANDLED EXCEPTION: System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. ---> System.Net.Sockets.SocketException: Connection reset by peer (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, System.Int32 offset, System.Int32 size, System.Net.Sockets.SocketFlags socketFlags) [0x00017] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referen
cesource/System/net/System/Net/Sockets/Socket.cs:1773  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x0009b] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/System/net/System/Net/Sockets/N
etworkStream.cs:513  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :    --- End of inner exception stack trace --- (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 size) [0x000c3] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/System/net/System/Net/Sockets/N
etworkStream.cs:525  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Net.RequestStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00033] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/System/System.Net/RequestStream.cs:141  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at MonoTests.System.Net.StreamExtensions.ReadAll (System.IO.Stream stream, System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00004] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at MonoTests.System.Net.HttpWebRequestTest+<>c__DisplayClass63_0.<PrematureStreamCloseAborts>b__1 (System.Net.HttpListenerContext c) [0x0000b] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at MonoTests.System.Net.HttpWebRequestTest+ListenerScope+<>c__DisplayClass4_0.<RequestHandler>b__0 (System.Object o) [0x00000] in <506591a0ad94482da7f4d89d0939a426>:0  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) [0x0000d] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1308  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:961  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:908  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00021] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1285  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00074] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:858  (TaskId:131)
  09-12 20:24:06.298  4797  7962 E mono-rt :   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /Users/builder/jenkins/workspace/xamarin-android/xamarin-android/external/mono/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1213  (TaskId:131)

@grendello grendello removed the do-not-merge label Sep 13, 2018

namespace EmbeddedDSOUnitTests
{
sealed class LocalBuilder : Builder

This comment has been minimized.

@jonpryor

jonpryor Sep 13, 2018

Contributor

I wonder if these unit tests should be integrated with/moved into src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests.

@jonpryor

jonpryor Sep 13, 2018

Contributor

I wonder if these unit tests should be integrated with/moved into src/Xamarin.Android.Build.Tasks/Tests/Xamarin.Android.Build.Tests.

This comment has been minimized.

@grendello

grendello Sep 13, 2018

Contributor

I kept them separate because the Android project used by those tests is also a standalone test in its own right (with instrumentation and such)

@grendello

grendello Sep 13, 2018

Contributor

I kept them separate because the Android project used by those tests is also a standalone test in its own right (with instrumentation and such)

Show outdated Hide outdated tests/RunApkTests.targets Outdated
#endif
TRY_LIBMONOSGEN (app_libdir)
if (!embedded_dso_mode) {

This comment has been minimized.

@jonpryor

jonpryor Sep 13, 2018

Contributor

This condition concerns me, in that it means that if a Debug build was using "embedded DSO support," that Debug build can't use Improved Fast Deployment, because we only look in the override directories when embedded_dso_mode==0.

@jonpryor

jonpryor Sep 13, 2018

Contributor

This condition concerns me, in that it means that if a Debug build was using "embedded DSO support," that Debug build can't use Improved Fast Deployment, because we only look in the override directories when embedded_dso_mode==0.

@grendello

This comment has been minimized.

@jonpryor

jonpryor Sep 14, 2018

Contributor

I tested this PR in Debug mode :)

Which is wonderful! The problem is that my "static analysis" (trying to read this PR) doesn't easily see why that works, not easily, not obviously. Here, within get_libmonosgen_path(), when embedded_dso_mode==TRUE, how do we find .__override__/.../libmonosgen-2.0.so? Do we find libmonosgen-2.0.so within this function at all, or is it found "elsewhere"? If it is "elsewhere," where is that "elsewhere"?

Turns out, that "elsewhere" is in Java_mono_android_Runtime_init():

	if (embedded_dso_mode) {
 		libmonosgen_handle = load_dso_from_any_directories (MONO_SGEN_SO, sgen_dlopen_flags);
 	}

I wonder if things would be clear if instead of a get_libmonosgen_path() we had a load_libmonosgen(), so that the "find libmonosgen" and "load libmonosgen" logic weren't so far apart. :-/

@jonpryor

jonpryor Sep 14, 2018

Contributor

I tested this PR in Debug mode :)

Which is wonderful! The problem is that my "static analysis" (trying to read this PR) doesn't easily see why that works, not easily, not obviously. Here, within get_libmonosgen_path(), when embedded_dso_mode==TRUE, how do we find .__override__/.../libmonosgen-2.0.so? Do we find libmonosgen-2.0.so within this function at all, or is it found "elsewhere"? If it is "elsewhere," where is that "elsewhere"?

Turns out, that "elsewhere" is in Java_mono_android_Runtime_init():

	if (embedded_dso_mode) {
 		libmonosgen_handle = load_dso_from_any_directories (MONO_SGEN_SO, sgen_dlopen_flags);
 	}

I wonder if things would be clear if instead of a get_libmonosgen_path() we had a load_libmonosgen(), so that the "find libmonosgen" and "load libmonosgen" logic weren't so far apart. :-/

This comment has been minimized.

@grendello

grendello Sep 15, 2018

Contributor

Yes, that would be better - this code could use a lot of improvements all over the map. However, I don't want to muddle the actual topic of this particular PR with logic and code cleanup. I'd like to make such changes/improvements after we merge in the C++ PR. I purposefully stayed as close to original code when implementing this feature as possible (even though I often felt like rewriting large portions of what I saw - but that would mean going down the rabbit hole and making this PR something that cannot be properly reviewed)

@grendello

grendello Sep 15, 2018

Contributor

Yes, that would be better - this code could use a lot of improvements all over the map. However, I don't want to muddle the actual topic of this particular PR with logic and code cleanup. I'd like to make such changes/improvements after we merge in the C++ PR. I purposefully stayed as close to original code when implementing this feature as possible (even though I often felt like rewriting large portions of what I saw - but that would mean going down the rabbit hole and making this PR something that cannot be properly reviewed)

@grendello grendello changed the title from [WIP] First steps to support DSOs embedded in the .apk to First steps to support DSOs embedded in the .apk Sep 13, 2018

@grendello grendello changed the title from First steps to support DSOs embedded in the .apk to Support DSOs embedded in the .apk Sep 13, 2018

@grendello

This comment has been minimized.

Show comment
Hide comment
@grendello

grendello Sep 13, 2018

Contributor

build

Contributor

grendello commented Sep 13, 2018

build

log_fatal (LOG_BUNDLE, "bundled app initialization error");
exit (FATAL_EXIT_CANNOT_LOAD_BUNDLE);
} else {
log_info (LOG_BUNDLE, "bundled app not found in the APK, ignoring.");

This comment has been minimized.

@jonpryor

jonpryor Sep 14, 2018

Contributor

Why should we ignore here and not exit? Why would embedded_dso_mode matter for this check?

@jonpryor

jonpryor Sep 14, 2018

Contributor

Why should we ignore here and not exit? Why would embedded_dso_mode matter for this check?

This comment has been minimized.

@grendello

grendello Sep 15, 2018

Contributor

Short version: if we exit, embedded DSO stops working.

Long version: in non-embedded mode the decision to set up a bundled app is based on the presence, on the filesystem, of the bundled app .so file - we stat the filesystem to make the decision. In embedded mode we cannot do the stat inside the zip/apk and so we always dlopen all the .so and see if that succeeded. Since it would always fail here, if the application wasn't a bundled one, the DSO mode would never work. As it currently stands, there's no other way to make this work except to ignore the failure to load the bundle DSO. It can (and should) be improved (as stated in the comment above the setup_bundled_app function) by allowing the code to detect if we're inside a bundled app by other means - for instance an environment variable added to the environment file when building the app. This change, however, is beyond scope of this PR.

@grendello

grendello Sep 15, 2018

Contributor

Short version: if we exit, embedded DSO stops working.

Long version: in non-embedded mode the decision to set up a bundled app is based on the presence, on the filesystem, of the bundled app .so file - we stat the filesystem to make the decision. In embedded mode we cannot do the stat inside the zip/apk and so we always dlopen all the .so and see if that succeeded. Since it would always fail here, if the application wasn't a bundled one, the DSO mode would never work. As it currently stands, there's no other way to make this work except to ignore the failure to load the bundle DSO. It can (and should) be improved (as stated in the comment above the setup_bundled_app function) by allowing the code to detect if we're inside a bundled app by other means - for instance an environment variable added to the environment file when building the app. This change, however, is beyond scope of this PR.

mono_mkbundle_init = dlsym (libapp, "mono_mkbundle_init");
if (!mono_mkbundle_init)
log_error (LOG_BUNDLE, "Missing mono_mkbundle_init in the application");

This comment has been minimized.

@jonpryor

jonpryor Sep 14, 2018

Contributor

Should this also exit the process?

@jonpryor

jonpryor Sep 14, 2018

Contributor

Should this also exit the process?

This comment has been minimized.

@grendello

grendello Sep 15, 2018

Contributor

Ditto :)

@grendello

grendello Sep 15, 2018

Contributor

Ditto :)

grendello and others added some commits Sep 3, 2018

Support DSOs embedded in the APK
Context: #1906
Fixes: #1906

Android API 23 introduced a new way of dealing with the native shared libraries
shipped in the APK. Before that API level, the libraries would be always
extraced and placed in the application data directory, thus occupying more space
than necessary. API 23 added a new manifest `<application>` element attribute,
`android:extractNativeLibs`, which if set makes Android not extract the
libraries to the filesystem. API 23 added a way to load those libraries directly
from the APK. In order to support that there are a few requirements which this
commit implements:

  * DSO (`.so`) files must be stored uncompressed in the APK
  * `<application android:extractNativeLibs="false"/>` must be present
  * DSOs in the APK must be aligned on the memory page boundary (the `-p` flag
    passed to `zipalign` takes care of that

This commit also implements `libmonodroid` suport for loading our DSOs directly
from the APK. This operation mode is enabled by the presence of the
`__XA_DSO_IN_APK` environment variable. This variable is inserted into the
application's environment by way of placing it in the environment file (a file
part of the XA project that is marked with the `AndroidEnvironment` build
action). In that mode, the DSOs are *no longer* looked up in the application
data directory but only in the override directories (if the APK is built in
Debug configuration) and in the APK itself.

Currently, in order to activate the above mode, one has to perform the following
actions manually:

  * Add the `android:extractNativeLibs="false"` attribute to the `<application>`
    element in the `Properties/AndroidManifest.xml` file

  * Add the following property to the project file:

      <AndroidStoreUncompressedFileExtensions>.so</AndroidStoreUncompressedFileExtensions>

  * Add an android environment file to the project with a line which says

      __XA_DSO_IN_APK=1

After that the application should work in the embedded DSO mode without
problems.

A couple of tests are provided to test building and execution of embedded DSO
application on device, as well as to validate the built APK.

@jonpryor jonpryor merged commit 95ca102 into xamarin:master Sep 17, 2018

1 of 3 checks passed

Ubuntu PR Build Build started for merge commit.
Details
macOS PR Build Build triggered for merge commit.
Details
license/cla All CLA requirements met.
Details

@grendello grendello deleted the grendello:issue1906 branch Sep 17, 2018

@@ -33,25 +33,18 @@ void monodroid_dylib_mono_free (struct DylibMono *mono_imports)
free (mono_imports);
}
int monodroid_dylib_mono_init (struct DylibMono *mono_imports, const char *libmono_path)
int monodroid_dylib_mono_init (struct DylibMono *mono_imports, void *libmono_handle)

This comment has been minimized.

@radekdoulik
@radekdoulik

radekdoulik Sep 18, 2018

Contributor

This comment has been minimized.

@radekdoulik

radekdoulik Sep 18, 2018

Contributor

I can update Java.Interop. I wonder though, whether we consider libmonodroid API as public, because it can break someone else too?

@radekdoulik

radekdoulik Sep 18, 2018

Contributor

I can update Java.Interop. I wonder though, whether we consider libmonodroid API as public, because it can break someone else too?

This comment has been minimized.

@grendello

grendello Sep 18, 2018

Contributor

When we migrate to C++ there will be more changes here. That code becomes a class and I added extern "C" versions of the functions - we can standarize/stabilize the API after the migration.

@grendello

grendello Sep 18, 2018

Contributor

When we migrate to C++ there will be more changes here. That code becomes a class and I added extern "C" versions of the functions - we can standarize/stabilize the API after the migration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment