Android: DemoHybridApp fails to build due to 'networkSecurityConfig'

[zwieblum]

I just tried to get DemoHybridApp compiling for android. It fails with this error message:

-code-gen:
[mergemanifest] Merging AndroidManifest files into one.
[mergemanifest] Manifest merger disabled. Using project manifest only.
     [echo] Handling aidl files...
     [aidl] No AIDL files to compile.
     [echo] ----------
     [echo] Handling RenderScript files...
     [echo] ----------
     [echo] Handling Resources...
     [aapt] Generating resource IDs...
     [aapt] /home/nik/.lambdanative/tmp_build/bin/AndroidManifest.xml:8: error: No resource identifier found for attribute 'networkSecurityConfig' in package 'android'
     [aapt] 
     [aapt] /home/nik/.lambdanative/tmp_build/bin/AndroidManifest.xml:8: error: Error: No resource found that matches the given name (at 'networkSecurityConfig' with value '@xml/network_security_config').
     [aapt] 

BUILD FAILED
/home/entwicklung/lambdanative/android-sdk-linux/tools/ant/build.xml:649: The following error occurred while executing this line:
/home/entwicklung/lambdanative/android-sdk-linux/tools/ant/build.xml:694: null returned: 1

Total time: 1 second
ERROR: failed with exit code 1
BUILD FAILED
make: *** [Makefile:2: all] Fehler 1

Any idea what to do about this?

[mgorges]

This is a substitution coming from modules/hybridapp/ANDROID_application_attributes, with the file included being in apps/DemoHybridApp/xml/network_security_config.xml, though my version of the latter is slightly different allowing cleartextTraffic only on localhost (which i added in 2e32b1b)? It was introduced in fc9cfce but I don't know why this file is not getting pulled into your package; the transfer happens in targets/android/build-binary#L245

[zwieblum]

I just checked: the xpm file is copied ~/.lambdanative/tmp_build/res/xml/network_security_config.xml in line 252 and is still there when make fails.

I stumbled across this, but don't know what to make of it:
https://stackoverflow.com/questions/41936253/error-in-configuring-network-security-xml-in-android#41936620

[mgorges]

I am building with an ANDROIDAPI=21 and haven't changed the target of android:targetSdkVersion="31". I am not sure about that reference as it mentions android/app/src/main/res/xml/network_security_config.xml and all we have would be src/org/ecemgroup/demohybridapp/*, presumably as we still build with ant and not gradle?

[mgorges]

Update: It seems from #440 (comment) that you are building for an ANDROIDAPI=23, so I would have to try that instead. Yet, only 23 but neither 21, 24 nor 31 exhibit this problem for me on macOS, so there might be something different about it? Thus, unless you must have 23, I suggest you try either 21 or 24 instead.

[zwieblum]

You are right. I changed to ANDROIDAPI=24 and the error is gone. Might be a good idea to put a red flag on 23 :)

There remains an issue though: The compiled apk is installable, but when run it complains about "localhost not found". Did something change in android that localhost is not resolved to 127.0.0.1 any more? If yes, then I'd suggest to use 127.0.0.1 in

./modules/hybridapp/ANDROID_java_variables:    mWebView.loadUrl("http://localhost:8080");

[mgorges]

I have added more IPs (10.0.2.2 and 10.0.3.2) to be whitelisted in 9be9dd5, and can make that change too dcbf8c0, but I wonder if that breaks things in the simulator?

[zwieblum]

IMO this should break nothing. I tried the android emulator on linux, but it does not work: the amd64 systemimage boots, but cannot run the arm64 programs. The arm64 systemimage does not boot at all. So without a real device debugging is a bit hard :)

[mgorges]

I believe you can make a targets/android/host_linux_x86_64 file with this content (based on macOS - I didn't try this):

SYS_CPU=x86_64
ANDROIDARCH=x86_64
SYS_ANDROID_ABI=x86_64
SYS_ANDROID_ABIs="$SYS_ANDROID_ABIs $SYS_ANDROID_ABI"
. $SYS_ROOT/targets/android/_host_linux

[zwieblum]

That's nice, this work on linux, too. I can upload DemoHybridApp. But when running it I get net::ERR_CONNECTION_REFUSED I do not know if this means that the native code part connected to a different IP or that there is a permission issue. I don't get any error message on the terminal from were I launched the emulator. DemoCube works just fine in the emulater.

[mgorges]

That sounds like a permission problem! Unfortunately, I don't know which part you refer to - does the app fail both on a physical device and the simulator? The 'localhost' IP for the desktop behind the simulator maybe 10.0.2.2, not 127.0.0.1 (as the emulator runs behind a virtual router - see https://developer.android.com/studio/run/emulator-networking). I don't have a physical android device to test with but might be able to look more into the simulator over the weekend.

[zwieblum]

Yes, both fail (real device and emulator) with the same error message. I tried:

• localhost
• 127.0.0.1
• 10.0.2.2

I do not connect to the sim from outside the sim, that would not work without "black magic". But the webview part of the hybrid app all should connect to localhost (sim loopback device) and the lambdanative part should do the same (connect to loopback device inside the sim). I just found that "localhost" is resolved to 127.0.0.1 in the sim (as one would expect) - not 10.0.2.2. That should still be the same on real devices. Leaves a permission problem as the root of all evil.

[zwieblum]

Looks like the lambdanative part of hybridapp does not get a listen socket:
adb shell --> ps --> demohybridapp is running.
adb shell --> netstat -tl --> no tcp server sockets.

[mgorges]

Interesting, I assumed that android.permission.INTERNET in modules/hybridapp/ANDROID_xml_permissions#L5 would have been sufficient for this, but maybe that also changed - will need to borrow a newer android device unless this can be debugged in the simulator.

[mgorges]

Not much progress, but it seems that on Android the thread made with make-safe-thread in website-serve is never called. This is at modules/website/website.scm#L272.

[zwieblum]

So (thread-start! ...) does not work on android any more?

[mgorges]

I don't know. I am looking for regressions going back to older versions of LambdaNative and the Android SDK, but I will have time to review this more next week. It could also be a simulator problem, but I don't have recent Android hardware to test with.

[zwieblum]

Well, the problem is the same on real hardware - but I don't know enough about android internals to debug that efficiently.

[mgorges]

Yes, the problem with threads not running on newer Android devices is the same on real hardware, and it seems to be the case regardless of API targets (I tried 24, 26, 31, based on 24) for Gambit 4.9.2. I have yet to identify where this regression happened as previous builds (with custom toolchains, targeting 28 based on 26) work fine.
Update: I was wrong, it works for 24 with a target of 31 if I use the older Gambit 4.7.9, but to switch to that requires scrubbing the entire cache for all platforms, so its painful to test!

[mgorges]

This patch gets the web server to connect ... It also seems Android doesn't like nested threads?

diff --git a/modules/website/website.scm b/modules/website/website.scm
index 0d879e7..6014239 100644
--- a/modules/website/website.scm
+++ b/modules/website/website.scm
@@ -52,7 +52,7 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   string-replace-substring
   exception->string log:exception-handler
   log-error log-system
-  make-safe-thread
+  make-safe-thread app:android?
   u8vector->base64-string
   system-directory system-pathseparator string-contains
 ))
@@ -128,9 +128,14 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
   (let ((accept-port (open-tcp-server (list server-address: address port-number: port reuse-address: #t))))
     (let loop () (let ((connection (read accept-port)))
         (if (not (eof-object? connection))
-            (begin (thread-start! (make-safe-thread (lambda ()
-              (current-exception-handler log:exception-handler)
-              (website:serve db connection) )))
+            (begin
+              ;; Android doesn't seem to like a thread in a thread?
+              (if app:android?
+                (website:serve db connection)
+                (thread-start! (make-safe-thread (lambda ()
+                  (current-exception-handler log:exception-handler)
+                  (website:serve db connection))))
+              )
      (loop)))))))
 
 (define (website:trim-string s)
@@ -267,9 +272,13 @@ SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 (define website:db (make-table))
 (define (website-getdb) website:db)
 
-(define (website-serve db port . bind) (thread-start! (make-safe-thread (lambda ()
-  (current-exception-handler log:exception-handler)
-  (website:server (if db db website:db) port (if (or (null? bind) (not (eq? (car bind) 'localonly))) "*" "127.0.0.1"))))))
+(define (website-serve db port . bind)
+  (thread-start! (make-safe-thread (lambda ()
+    (current-exception-handler log:exception-handler)
+    (website:server (if db db website:db) port (if (or (null? bind) (not (eq? (car bind) 'localonly))) "*" "127.0.0.1")))))
+  ;; It is unclear what this does, but it is needed on Gambit 4.9.2 to run on newer Android devices
+  (if app:android? (thread-sleep! 0.01))
+)
 
 (define (website-addhook db document proc)
   (table-set! (if db db website:db)

[mgorges]

See if 74f3a10, which implements the comment above, helps; otherwise, please suggest alternatives, but the root cause is threads are not bing activated or are hanging. The issue is limited to Gambit 4.9.2 (libgambit) but it works flawlessly with Gambit 4.7.9 (libgambc) regardless of the patch.

[zwieblum]

Well, mixed success with gambit 4.7.9: - DemoHybridApp compiles and works inside the x86 emulator - Democube compiles but does not work inside the x86 emulator (black screen) - My own Demoapplication (that uses some threads) compiles and the WebView connects to the local server, but the screen stays blank/white with header when run inside the x86 emulator --> only one thread is running (the webserver) but it is not fully functional and no other threads are running. I remember not too long ago I jumped to gambit 4.9.2 due to the old gambit not building on arm64. So IMO gambit 4.7.9 is not really an option if your hardware is not ancient.

…

-- Please do not email me anything that you are not comfortable also sharing with the NSA, CIA ...

[zwieblum]

This github 2FA is getting on my nerves, I will quit using my account. Is there an other way to send issues and comments? As for now I 've seen these 2 lines:

+  ;; It is unclear what this does, but it is needed on Gambit 4.9.2 to run on newer Android devices +  (if app:android? (thread-sleep! 0.01))

Android suspends processes and threads if they don't yield to the OS on regular intervals. IMO this is done in lambdanative with the heartbeat function, but this only works for the thread that calles this function. All other theads will have to do the same call or be suspended. Might be the cause of this problem.

…

-- Please do not email me anything that you are not comfortable also sharing with the NSA, CIA ...

[mgorges]

I am sorry to hear about your struggles with Github. Currently there is no alternative to this process here, as the discussion board we had on Linear was merged into Github too, and email doesn't scale well.

As for the sleeping piece, it is a hack but happens to work, so until we get better solutions, we have to use trial and error here. I don't use Android a lot so my abilities to troubleshoot and develop on it are limited - sorry.

[zwieblum]

Now this is OT, but a forum would most likely be a better way to build community than the github issue tracker. At least it would be searchable in a sane way (that is, if it's not disourse forum). At the moment I can't get lambdanative working for iOS devices at all (the other issue) or android (hybridapp), which is quite a pitty. The reasion to jump for hybridapp in the first place was that it works around the GUI rendering & layout issues the GL applications suffer - and that the webview offers all the bells and whistles you would have to build glue layers otherwise, i.g. live camera feed.

…

-- Please do not email me anything that you are not comfortable also sharing with the NSA, CIA ...

[mgorges]

There is https://github.com/part-cw/lambdanative/discussions. As far as I know hybrid apps work on Android and other than gambit not compiling on an arm mac with the latest macOS I am not aware of other iOS building problems either.

[zwieblum]

@ discussion: that still ues guthub 2FA ... Anyway, I think I have found the problem (gambit 4.9.2 - but should be the same for 4.7.*): "new" android webview does not allow page reloads, neither from javascript nor from <meta ...>. So everything that would like the page (or part of it) be reloaded fails. Usually there would be some debug output, but don't ask me where android sends that. Looks like this is the solution, but where should thi go? https://stackoverflow.com/questions/34979010/reloading-a-webview-with-a-javascript-call-from-a-loaded-web-page

…

-- Please do not email me anything that you are not comfortable also sharing with the NSA, CIA ...

[mgorges]

Yes, we used to have the separate https://spectrum.chat/lambdanative became Github Discussions in 2018. If there is a different community we should try I am happy to do give it a shot - but I can't dedicate a lot of time building a community elsewhere, sorry.

Logfiles on Android appear in the /sdcard/Android/data/[SYS_ORGSLD_REVERSE]/files/log (you might have to create the log folder) as that is the most accessible location on Android devices - see 212ab34. The link you provide is a new feature to reload the entire WebView (and the backend?) not just refresh a page - I think the example in one of your previous issues that had the counter worked though?

[zwieblum]

Oh, and there is most likely a second cops buried: ``` $ adb logbridge [...] cr_AwContentsClient: Denied starting an intent without a user gesture, URI http://127.0.0.1:8080/ ``` The proposed solution is here - again, I don't know where to put this: https://stackoverflow.com/questions/33048945/denied-starting-an-intent-without-a-user-gesture-webview-android

…

-- Please do not email me anything that you are not comfortable also sharing with the NSA, CIA ...

[mgorges]

https://github.com/part-cw/lambdanative/blob/master/modules/hybridapp/ANDROID_java_oncreate#L8 or anywhere below could work with something like this (the hybridapp webview has a slightly different name than the example):

mWebView.setWebViewClient(new WebViewClient() {
    @Override
    public boolean shouldOverrideUrlLoading(WebView view, String url) {
        return false;                
    }
});

Nevermind - this is not a setting, you want to do this dynamically, I believe so we'd have to wrap that into a scheme function?

[zwieblum]

I think the example in one of your previous issues that had the counter worked though?

Yes, that's how I found out. I don't know why in DemoHybridApp the call to CGI ```get_async(ajax_cmd, function(data) { ... }``` works. Looks like only a subset of JS functions / meta tags make the HybridApp dysfunctional.

…

-- Please do not email me anything that you are not comfortable also sharing with the NSA, CIA ...

[zwieblum]

Ok, looks this is a viable workaround. Basicly you must not load anything that has a <html> or <bod<> tag, you must not use reload, you must not tinter with the history ... there might be more hidden gotchas, but for now this is a way how to work around the "features" of webview. It boils down to: - load a minimal html page - make a body or div with and id - replace the inner html art of said div/html from javascript - don't ever use anything that reloads the page I'm not sure if you are allowed to use links or if there are sideeffects, too. Now this is an odd way to create web applications, but I remember there was a problem in older weview applications that caused the addressbar to appear when going to an other page (there was a workaround, too, IMO I posted it ther some years ago). Please see the attachned code for details.

…

-- Please do not email me anything that you are not comfortable also sharing with the NSA, CIA ...

[zwieblum]

< sigh >
DemoWebTemplate.zip