Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No more exec from data folder on targetAPI >= Android Q #1072

Open
n0n3m4 opened this issue Mar 18, 2019 · 77 comments
Labels

Comments

@n0n3m4
Copy link

@n0n3m4 n0n3m4 commented Mar 18, 2019

The ability to run execve() on files within an application's home directory will be removed in target API > 28.
Here is the issue on Google bug tracker:
https://issuetracker.google.com/issues/128554619
As expected it is yet another "working-as-intended", furthermore, as commented in the issue,

Relying on exec() may be problematic in future Android versions.

This seems to completely break Termux in a long-term, as all its packages contain executables.

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Mar 18, 2019

This definetly the worst thing that may happen.
Though, in AVD with Android Q Preview Termux still works:

Screenshot_1552922384

Screenshot_1552922415

@xeffyr xeffyr added the information label Mar 18, 2019
@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 18, 2019

I brought up that this was forbidden and would likely become part of the SELinux policy last year (9 Apr 2018): #655.

As I stated then,

They rewrote the policy to be much more explicitly about this being disallowed, and it can be expected that they'll not only act on it for the Play Store but via SELinux policy.

They intend to support cases like Termux by having apps like Termux upload their packages as separate apps to the Play Store, with it initiating user installs / uninstalls of those apps.

Staying ahead of the game (i.e. future SELinux policy enforcement) would involve not extracting the executables manually but rather running them from where the apk package installer extracts libraries. Being able to execute app_data_file either as an executable or a library is on the way out in the long term.

It doesn't fundamentally break Termux, but rather Termux would need to switch to an alternative approach for distributing packages. Using apt within the app sandbox to install code isn't going to be viable anymore. The restrictions are no doubt going to be tightened up in the future too, and it can be expected that libraries will need to be mapped from the read-only app data directory or apk too, rather than the read-write home directory working. It wouldn't be necessary to split absolutely everything into assorted apks, but rather there could be apks for groups of related packages based on size. Termux can initiate the apk installs itself too, after prompting the user to allow it as an unknown source. It's an inconvenience for sure, but not a deal breaker.

Someone there accused me of concern trolling, but I was genuinely giving an insider heads up about likely changes on the horizon bringing w^x enforcement for data in storage. I use Termux and also work on Android OS security, so I wanted to have a conversation about how this could be addressed long before it was going to become a problem. I provided enforcement of the same kind of restrictions in my own downstream work, but with a toggle to turn it off for apps like Termux. I would have loved to have an alternate option for installing packages compatible with the security feature, and that would have prepared Termux for the eventual enforcement of this feature upstream.

It's worth noting that there are still holes in the SELinux policy, which can be used to bypass it by jumping through some hoops, but the intention is clearly to move towards enforcing w^x and enforcing the official rules via the implementation. See https://play.google.com/about/privacy-security-deception/malicious-behavior/.

The following are explicitly prohibited:

[...]
Apps or SDKs that download executable code, such as dex files or native code, from a source other than Google Play.
[...]

Termux could also ship the necessary scripting for generating a Termux package apk from executables/libraries so that users could create their own and continue running their own code, albeit less conveniently.

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 18, 2019

@xeffyr This only applies to apps targeting API > 28. Termux will continue to work indefinitely as long as it continues to target API 28. However, there's a timeline for phasing out old API targets on the Play Store about a year after they're introduced. It then becomes impossible to upload a new version of the app without updating the target API level.

https://developer.android.com/distribute/best-practices/develop/target-sdk

Distributing the app outside the Play Store would still be possible, but the OS has the concept of a minimum supported target API level already (ro.build.version.min_supported_target_sdk). It's currently 17 in Android P and is being increased to 23 in Android Q. It currently only produces a warning, but it won't necessarily remain that way. They don't want targeting an old API level or distributing outside the Play Store to be a loophole for bypassing privacy and security restrictions tied to API level.

@Grimler91

This comment has been minimized.

Copy link
Member

@Grimler91 Grimler91 commented Mar 18, 2019

@thestinger

Someone there accused me of concern trolling

I apologize for how issue #655 was received, I guess people freaked out and decided to shoot the messenger. We should have moderated it better.

It's currently 17 in Android P and is being increased to 23 in Android Q

I guess we have a year or two (at least) to figure out how to deal with this then

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 18, 2019

You can probably stay at API 28 until around November 2020 for the Play Store assuming the timeline at https://developer.android.com/distribute/best-practices/develop/target-sdk is followed again for API 29.

It would be even longer until using API 28 broke outside the Play Store via a stricter future version of ro.build.version.min_supported_target_sdk doing more than warning. Probably at least 2 years and likely significantly longer.

So, it's not like it's going to suddenly break when Android Q comes out, since it's only for API 29+.

@its-pointless

This comment has been minimized.

Copy link

@its-pointless its-pointless commented Mar 19, 2019

@thestinger

Termux could also ship the necessary scripting for generating a Termux package apk from executables/libraries so that users could create their own and continue running their own code, albeit less conveniently.

That is horrible in terms of usability. Also wouldn't the apks need to be signed with original apk key?

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Mar 19, 2019

Not only signed with original key. It seems that executables will have to be placed to native lib directory:

While exec() no longer works on files within the application home directory, it continues to be supported for files within the read-only /data/app directory. In particular, it should be possible to package the binaries into your application's native libs directory and enable android:extractNativeLibs=true, and then call exec() on the /data/app artifacts. A similar approach is done with the wrap.sh functionality, documented at https://developer.android.com/ndk/guides/wrap-script#packaging_wrapsh .

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Mar 19, 2019

Termux compiled with targetSdkVersion 29:

Termux not working on Android Q


Exec is allowed from native lib directory:
Binaries in /lib folder on Android Q
But this directory is read-only.

Distributing packages in APKs will either require to have hundreds of these APKs published to the Google Play / F-Droid or put multiple packages into single APK up to maximal size (~ 100MB). Technically it is possible to have very large APKs, but they cannot be distributed with Google Play and maybe F-Droid.

But anyway, distributing all packages in APKs is weird:

  • Such distribution may be a bit confusing from user's side.
  • Much longer waiting for updates, in case if distribution is done from Google Play.

Also, we should not forget that user environment will be broken:

  • It won't be possible to compile stuff in the Termux.
  • If somehow managed to compile, this stuff will be useless anyway.
  • Most stuff installable with pip, gem, cpan will be broken.
@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 19, 2019

Python, Ruby, Perl, etc. written by users will still work without a hassle because it's not native code. They won't be able to execute it as ./script.py but they can still run it as python3 script.py. I don't see any way that would ever stop working. Libraries without native code are also going to continue working without being in the native library directory too, since they don't get mapped as executable.

In the future, native libraries will likely all need to be in the native library directory to work but for API 29 they're only enforcing it for executables. I expect the enforcement will happen for native libraries next year, since they already added an auditallow rule to warn in the logs when it occurs.

Distributing packages in APKs will either require to have hundreds of these APKs published to the Google Play / F-Droid or put multiple packages into single APK up to maximal size (~ 100MB). Technically it is possible to have very large APKs, but they cannot be distributed with Google Play and maybe F-Droid.

Yes, that's what I was saying would need to happen: bundling groups of related packages inside of shared uid apks signed with the same key. For users to be able to extend it with their own native code, they would need to have their own build of Termux signed with their own key, along with scripting for generating new extension apks out of their code. There might be a better approach than my suggestion. It was just intended as a starting point.

It will also still be possible to interpret native code from the app data directory. For example, I think valgrind executable_in_app_data will continue working, even though ./executable_in_app_data will not, since valgrind acts as a machine code interpreter. Similarly, running native code inside QEMU would obviously still work.

@caleb-allen

This comment has been minimized.

Copy link

@caleb-allen caleb-allen commented Mar 19, 2019

Is this a potential use case for Dynamic Delivery using App Bundles?

A dynamic feature module could be either a bundled group of related packages, or, if still feasible, each separate package. The submitted app bundle would include all available packages, where the base installation is only core packages, and pkg install would request the dynamic feature from Google Play.

This would likely require a pretty robust build system to keep the Google Play listing's packages in parity with termux-packages, perhaps as a separate listing from termux-core.

I see this as a way to comply with Google Play while still allowing a package manager-like function, potentially with another mechanism for installations outside of Google Play or from source that more closely resembles the current mechanism.

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 19, 2019

Is this a potential use case for Dynamic Delivery using App Bundles?

It could work, if dynamic feature modules are able to ship additional executables / libraries to be extracted into the native libraries directory. It's not clear if that's possible. It appears the limit is 500MB via an app bundle, rather than 100MB, so it's a limited solution:

Publishing with Android App Bundles also increases the app size limit to 500MB without having to use APK expansion files. Keep in mind, this limit applies only to the actual download size, not the publishing size. So, users can download apps as large as 500MB and, with dynamic delivery, all of that storage is used for only the code and resources they need to run the app. When you combine this with support for uncompressed native libraries, larger apps, such as games, can reduce disk usage and increase user retention.

I see this as a way to comply with Google Play while still allowing a package manager-like function, potentially with another mechanism for installations outside of Google Play or from source that more closely resembles the current mechanism.

I don't think it's going to be able to be much different outside the Play Store, since the SELinux policy restrictions on executing only read-only native code still apply for apps outside the Play Store and Android variants without the Play Store or Play Services. It would make sense to work around this issue at the same time as coming into compliance with the Play Store policy on downloading executable code, but they don't necessarily need to be solved at the same time. I think it would be best to choose a path that will work both inside and outside the Play Store. I don't know how realistic it is to support the app bundle and dynamic feature modules outside Play, since there isn't an existing distribution option with support for it and it's not clear how much is available without Play Services, etc.

@caleb-allen

This comment has been minimized.

Copy link

@caleb-allen caleb-allen commented Mar 19, 2019

Sure--I supposed my suggestion was intended more as a solution to the SELinux policy restrictions by using Google Play, but it does hinge on 1) the ability to ship additional executables (I suspect their example of games would make that true, but it is still a question) and 2) reliance on Google Play as a complete distribution system and being subject to their rules and whims.

It's a difficult problem, no doubt. I suspect that the timing of App Bundles and the native executable restriction policy are not entirely unrelated, and a solution to downloading and executing arbitrary native code without going through Google Play isn't immediately obvious.

bundling groups of related packages inside of shared uid apks signed with the same key.

Seems to be the quickest and simplest solution at this point, with the above mentioned restrictions which may annihilate realistic usability.

@corbinlc

This comment has been minimized.

Copy link

@corbinlc corbinlc commented Mar 19, 2019

What if you bundled PRoot (and busybox and a bare minimum of what is required to get a PRooted system setup and running) with the app itself and then had everything run through PRoot. Could PRoot be modified to allow executing outside of the lib directory? It has been proven twice before to be able to get around limitations of the sdcard being mounted noexec.

This is the most recent example of how PRoot can defeat noexec: termux/proot@ebcfe01

Notice what is not blocked here: https://android-review.googlesource.com/c/platform/system/sepolicy/+/804149

There would be a performance impact of course, but maybe it allows the party to go on.

I work on UserLAnd and we always use PRoot. This is where my mind went when I heard of this limitation. Thought I would share.

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Mar 19, 2019

If proot works, this means that we can left packages as-is. Proot also means that Termux will no longer be prefixed which is good as a lot of patches can be dropped + less headache with compiling software by user.

Will try to package proot into Termux APK and run on Android Q AVD to test this.

@n0n3m4

This comment has been minimized.

Copy link
Author

@n0n3m4 n0n3m4 commented Mar 19, 2019

I think that proot (or other theoretically possible packages like this) will be broken no earlier than Google forbids any kind of JIT in 3rd-party applications, so this is a very nice idea. I'm still not sure if Google will ever dare to do such thing.

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Mar 19, 2019

forbids any kind of JIT

If Google do this, it has to forbid javascript in browsers too.

@corbinlc

This comment has been minimized.

Copy link

@corbinlc corbinlc commented Mar 19, 2019

PRoot will certainly execute from the apps lib directory, just like busybox did above, but you will need to play with the proot version I mentioned above. It is @michalbednarski 's work, so he might be able to comment on how suited he thinks it might be for this purpose.

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Mar 19, 2019

@corbinlc Original proot worked. Even this anti-noexec extension not needed.

Proot 1

So under proot everything works fine:
Proot 1

No need to distribute packages in APKs or bundles. Everything that is needed is to launch shell under proot.

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 19, 2019

Notice what is not blocked here: https://android-review.googlesource.com/c/platform/system/sepolicy/+/804149

Note that execute for app_data_file (mapping as PROT_EXEC) had an auditallow rule added after this commit as they're planning on removing that in the near future, perhaps next year. It will currently log audit warnings leading up to the likely eventual removal.

The remaining holes in the SELinux policy are tied to the ART JIT and other JIT compilers: ashmem execute (an implementation detail that could change) and execmem (in-memory rwx code and rw -> rx transitions). ART supports using full AOT compilation instead of the JIT compiler, but it's not their current choice.

Chromium-based browsers including the WebView use execmem in the isolated_app domain, not untrusted_app, so even without an Apple-like approach only permitting the bundled browser and WebView for other apps to use, that wouldn't be a direct blocker. I would be surprised if they removed execmem for untrusted_app in the near future, but it could happen over the longer term, especially if they simply restrict it to a variant of isolated_app.

The original issue that I filed (#655) was about the general problem of this being in violation of their policy, before there was a partial implementation of a technical mechanism enforcing it for native code. There can never be full enforcement at a technical level since interpreters will continue to work even once native code execution is forbidden.

If Google do this, it has to forbid javascript in browsers too.

They permit JavaScript via the exception in the policy for isolated virtual machines:

An app distributed via Google Play may not modify, replace, or update itself using any method other than Google Play's update mechanism. Likewise, an app may not download executable code (e.g. dex, JAR, .so files) from a source other than Google Play. This restriction does not apply to code that runs in a virtual machine and has limited access to Android APIs (such as JavaScript in a webview or browser).

https://play.google.com/about/privacy-security-deception/malicious-behavior/

The result at a technical level would be that they need to keep around execmem for a variant of the isolated_app SELinux domain used by browsers, but not untrusted_app. At the moment, both isolated_app and untrusted_app permit execmem, but Chromium doesn't use it in untrusted_app. ART uses the untrusted_app execmem when configured to do JIT compilation via the relevant system properties. It's fully optional though, and still supports full AOT compilation as was done in Android 6.

@corbinlc

This comment has been minimized.

Copy link

@corbinlc corbinlc commented Mar 19, 2019

@xeffyr I guess that makes sense, since PRoot already deals with execve. The new work is only needed for files on a noexec partition. That is great news for everyone.

@corbinlc

This comment has been minimized.

Copy link

@corbinlc corbinlc commented Mar 19, 2019

@thestinger interesting comment about them potentially removing the ability to mmap(PROT_EXEC) for files outside of the app's lib directory in the future. I guess we live to fight another day (or more likely more than a year).

Do you know if they are planning on preventing mprotect(PROT_EXEC)? Where do you see the audit call you mentioned related to mmap(PROT_EXEC)? I don't know where the these exist in the Android source tree.

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 19, 2019

Files in the app home directory are labelled as app_data_file and this is the audit rule generating warnings for mapping the data as executable:

https://android.googlesource.com/platform/system/sepolicy/+/08450264ae3f917f6b8e4091d6fedf84ef8d796f/private/untrusted_app_all.te#27

They add auditallow rules leading up to eventually removing it from the policy. You can try loading a library from app data with API 29 and you'll see an audit warning in the logs.

In-memory code generation via rwx pages or rw -> rx transitions is classified as execmem, which is something they'd like to remove in the long-term but it's currently used by the ART JIT. It's not used by Chromium within untrusted_app domains, since it only does that within the sandboxed renderers which use isolated_app domains. They could remove it from the main isolated_app and require that apps opt-in to having it available via a special domain. It should be expected that it will be eventually happen, since this is a very standard form of security hardening and they've been slowly moving towards it.

They already removed execmem for some of the base system including system_server. Their security team would obviously like to remove it everywhere, but for now the performance vs. memory vs. storage usage characteristics of mixed AOT and JIT compilation for ART are winning out over the security advantages of pure AOT compilation with execmem disallowed.

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 19, 2019

The only reason they're starting with requiring executables to be in the native library directory rather than native libraries (which is the main purpose of the directory) is because that will impact more apps, so the transition will be more painful. Apps can also disable native library extraction and map native libraries from the apk directly, since the linker supports mapping libraries from zips directly as long as they meet the requirements (page aligned and not compressed). They have a pass for properly aligning them in the apk build system.

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 19, 2019

It will actually log the warning about native libraries in the data directory for all API levels.

Here's where they continue to allow running executables from app_data_file for API < 28:

https://android.googlesource.com/platform/system/sepolicy/+/08450264ae3f917f6b8e4091d6fedf84ef8d796f/private/untrusted_app_27.te#30

Even though it's allowed, they do still log audit warnings about it for API < 29.

Also you can see that for API < 26, even more is permitted, since they've tightened up these policies for API 26+ in the past too:

https://android.googlesource.com/platform/system/sepolicy/+/08450264ae3f917f6b8e4091d6fedf84ef8d796f/private/untrusted_app_25.te

For example, API 26+ forbid execmod which means things like rx -> rw -> rx transitions for files mapped from storage. You can only do those kinds of transitions in-memory via execmem now. They've removed execmem for a lot of the base system but not yet apps, which is as I mentioned primarily due to the ART JIT and it'd also be a major backwards incompatible change. Finishing this up by removing the remaining 3 or so holes in the policy is definitely on their radar. It's just hard to get it done due to compatibility issues and performance/memory/storage vs. security tradeoffs made elsewhere that they'd need to decide are less important (and I definitely think they will, eventually).

@corbinlc

This comment has been minimized.

Copy link

@corbinlc corbinlc commented Mar 19, 2019

This is great info @thestinger

@esminis

This comment has been minimized.

Copy link

@esminis esminis commented Mar 23, 2019

One more thing to mention Google is planning to deny any type of exec, read here: https://issuetracker.google.com/issues/128554619

Relying on exec() may be problematic in future Android versions.

The steps they took:

  1. Deny downloaded code execution using policy
  2. Now denying execution from /data/data/[app]
  3. Next step would probably be denying any exec, removing Runtime.exec / ProcessBuilder, ...

So any solution implemented here will probably not work next year or even earlier. Essentially they will kill apps like Termux, server apps, ... unless you build everything using jni / ndk.

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Mar 23, 2019

Essentially they will kill apps like Termux, server apps, ... unless you build everything using jni / ndk.

Not kill, if we switch to QEMU (system mode). It should be possible to turn QEMU into shared library (Limbo PC app did this) and attach serial consoles as terminals. Performance will be worse, but still usable.

That is, of course, if QEMU doesn't use mmap(PROT_EXEC), execmem or other similar things which may be blacklisted.

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Mar 23, 2019

One more thing to mention Google is planning to deny any type of exec

That's not backed up by what you're linking and I don't think it's true.

Deny downloaded code execution using policy

Executing downloaded code was denied by policy for a long time too. I only opened an issue when the wording became substantially stronger and considered malicious regardless of why it's done. I think Termux has been in violation of the policy as long as it has existed, but it became more clear that it was serious a year ago.

Next step would probably be denying any exec, removing Runtime.exec / ProcessBuilder, ...
So any solution implemented here will probably not work next year or even earlier. Essentially they will kill apps like Termux, server apps, ... unless you build everything using jni / ndk.

So any solution implemented here will probably not work next year or even earlier. Essentially they will kill apps like Termux, server apps, ... unless you build everything using jni / ndk.

Spawning processes with exec based on the native library directory is fully allowed by their app policy and documented as a supported feature. A random Google engineer on the bug tracker urging you to be cautious about using exec is good advice and doesn't imply there's any plan to remove it. Mapping code from outside the native library directory and apk as exec is definitely on the way out, and class loading is probably on their radar. There is not much they could do about third party interpreters beyond forbidding it in policy. The general policy is that executing arbitrary code not included with the apk is only allowed if it runs in a well sandboxed virtual machine, with the Chromium web sandbox as a baseline to compare since it's allowed through that exception.

They have no security reason to remove it now that they enforce w^x for it. Relying on executing system executables is a bad idea and it'd make sense for them to continue stripping down what is allowed.

By saying it's problematic, they are not saying that it's going to be forbidden, but rather that it doesn't fit well into a model where apps are often killed at any time and then respawned with the expectation that they saved all of their state to be restored, etc. That model can be more aggressive on some devices, which is what makes it problematic. Switching away from the app can result in all these native processes getting killed, potentially losing data. In practice, it's fine, especially with a foreground service to prioritize keeping the app alive. It also applies to a lot more than native processes. A long-running foreground service doesn't conform to the activity life cycle either. Essentially the same potential for problems there.

Not kill, if we switch to QEMU (system mode). It should be possible to turn QEMU into shared library (Limbo PC app did this) and attach serial consoles as terminals. Performance will be worse, but still usable.

I also don't think it's true.

That is, of course, if QEMU doesn't use mmap(PROT_EXEC), execmem or other similar things which may be blacklisted.

I think QEMU does rely on execmem. I'm not sure that it can be used without it. So, while that will currently work, it may not in the long term. I do think executing files from the native library directory is going to continue working in the long term. Using an emulator is certainly a viable approach though, but obviously slow, especially without execmem at some future point.

The viable long-term approach is using documented ways of doing things, and they have documentation showing how to execute something from the native library directory, along with that being fully supported for native libraries.

@esminis

This comment has been minimized.

Copy link

@esminis esminis commented Mar 23, 2019

One more thing to mention Google is planning to deny any type of exec

That's not backed up by what you're linking and I don't think it's true.

Well I wish it was not not true but, doubt it following recent actions by Google / Android.

Deny downloaded code execution using policy

Executing downloaded code was denied by policy for a long time too. I only opened an issue when the wording became substantially stronger and considered malicious regardless of why it's done. I think Termux has been in violation of the policy as long as it has existed, but it became more clear that it was serious a year ago.

It was denied by policy not so long ago - a year or two.

They have no security reason to remove it now that they enforce w^x for it. Relying on executing system executables is a bad idea and it'd make sense for them to continue stripping down what is allowed.

So essentially strip down everything and allow only some games, ad/spam-ware apps to run, that would be really secure. w^x does not change security in most cases on Android cause most of security/privacy issues are created by pure Java / Kotlin apps.

By saying it's problematic, they are not saying that it's going to be forbidden, but rather that it doesn't fit well into a model where apps are often killed at any time and then respawned with the expectation that they saved all of their state to be restored, etc. That model can be more aggressive on some devices, which is what makes it problematic.

"problematic in future" - means that it is not problematic now, so I doubt they are saying what you mentioned here

The viable long-term approach is using documented ways of doing things, and they have documentation showing how to execute something from the native library directory, along with that being fully supported for native libraries.

You mean "problematic in future" Runtime.exec? :)

I think the only viable in future approach is compile everything using NDK and execute using JNI, but in light of Fuschia / Chrome OS changes it is very unclear.

@Grimler91

This comment has been minimized.

Copy link
Member

@Grimler91 Grimler91 commented Mar 25, 2019

I'll mark future comments (and some of the previous comments) that are about philosophical topics (in my opinion) rather than about the android Q security enhancement and what it means for termux as spam.

@TheBrokenRail

This comment has been minimized.

Copy link

@TheBrokenRail TheBrokenRail commented Apr 18, 2019

Is this a potential use case for Dynamic Delivery using App Bundles?

Android App Bundles seem to have an issue where they force extractNativeLibs to be false.

@esminis

This comment has been minimized.

Copy link

@esminis esminis commented Apr 18, 2019

Is this a potential use case for Dynamic Delivery using App Bundles?

Android App Bundles seem to have an issue where they force extractNativeLibs to be false.

Also I wonder about app bundle max size, I think it is 500 Mb

@TheBrokenRail

This comment has been minimized.

Copy link

@TheBrokenRail TheBrokenRail commented Apr 21, 2019

You can force extractNativeLibs=true in an Android App Bundle, by placing android.bundle.enableUncompressedNativeLibs=false into gradle.properties.

@caleb-allen

This comment has been minimized.

Copy link

@caleb-allen caleb-allen commented Apr 21, 2019

With a 500 Mb max, you might still be able to list multiple groups of packages as separate listings (yielding multiple packages below 500 Mb total). However, this would still require another solution for distribution outside Google Play

@xeffyr xeffyr mentioned this issue Aug 25, 2019
@xeffyr xeffyr pinned this issue Sep 8, 2019
@corbinlc

This comment has been minimized.

Copy link

@corbinlc corbinlc commented Oct 11, 2019

For what it is worth, we updated UserLAnd a while back to include proot, busybox and everything else needed for getting launched into a rootfs as libraries within the APK. I can confirm that this method made us Q compatible and our last several releases have targeted the latest SDK.
This method should then definitely work for Termux, but it will have some pluses and minuses. The main negative is that there will be a performance hit.

@kenneth-Q

This comment has been minimized.

Copy link

@kenneth-Q kenneth-Q commented Oct 23, 2019

Samsung to Dex is dead.
https://www.xda-developers.com/samsung-ends-linux-on-dex-beta-android-10-update/amp/
We have only termux can use. I am sadly to heard the android 10 of exec issue.
I use termux for java development. OMG

@caleb-allen

This comment has been minimized.

Copy link

@caleb-allen caleb-allen commented Oct 24, 2019

There is some new info from Google's Android Developer Summit.

https://www.youtube.com/watch?v=JgpCrYkC1OM

Here, Google is discussing changes in the storage API, the negative feedback some of it got, and what they are doing moving forward.

I was surprised to see the bullet point

  • Support for native libraries and file paths

Screen Shot 2019-10-24 at 10 38 28 AM

There are many more changes besides just that, some possibly related.

Screen Shot 2019-10-24 at 10 41 11 AM

Some points of particular interest to this issue, for the next Android release

Enable file paths and native libraries for media
Enforcement by target SDK

What does native libraries for media mean? They did not expand on that.

Nothing is said for execution restrictions, outside of the bullet point, and a brief mention that "many developers have the need for C libraries".

There is still much ambiguity. But could this be a signal that Termux can target new versions of Android?

@caleb-allen

This comment has been minimized.

Copy link

@caleb-allen caleb-allen commented Oct 24, 2019

Also possibly related is a permission for unrestricted file system access.

We're also adding special app access for apps which can truly demonstrate a need for broad access to shared storage. There are a limited number of use cases that we believe can demonstrate the need. A couple examples are file manager and back-up and restore apps. Only apps that can prove this need will be given access to it.

Screen Shot 2019-10-24 at 11 01 22 AM

@M4rtinK

This comment has been minimized.

Copy link

@M4rtinK M4rtinK commented Oct 24, 2019

Only apps that can prove this need will be given access to it.

I really wonder how they want to handle this in practice - hopefully not in the same way iOS App Store QA forces developers to change their apps in arbitrary ways to get them approved.

@caleb-allen

This comment has been minimized.

Copy link

@caleb-allen caleb-allen commented Oct 24, 2019

It could also be interpreted as media access via native libraries (with the NDK), in which case this doesn't change anything

@caleb-allen

This comment has been minimized.

Copy link

@caleb-allen caleb-allen commented Oct 24, 2019

https://android-review.googlesource.com/c/platform/bionic/+/954688

From June 10. Could be what was alluded to.

Add android_run_executable()

android_run_executable() works similar to execv(), but with an
important difference that it preserves existing memory mappings.
@cgarz

This comment has been minimized.

Copy link

@cgarz cgarz commented Oct 24, 2019

Only apps that can prove this need will be given access to it.

Thank you Google for making my decisions for me. I hate having to the make the effort to decide things for myself.

@bndeff

This comment has been minimized.

Copy link

@bndeff bndeff commented Nov 19, 2019

For anyone following this issue, note that there has been some relevant discussion on gitter in the last two weeks:
https://gitter.im/termux/dev

See also:
https://wiki.termux.com/wiki/Dev:Development_Sessions
https://github.com/termux/termux-packages/wiki/Termux-and-Android-10/071b6335ac530601b1d0b321ebb8d1cb23491a71
https://github.com/termux/termux-packages/wiki/Termux-and-Android-10/51dfe08a7e54351ea11c2994a6bc10b492bd818f

By the way, did you consider and rule out the suggestion by @n0n3m4 and @thestinger to use isolatedProcess or was it just lost in the comment thread? It's the method used by Chrome to run web code in a sandbox and Firefox Preview is moving to it as well. And it seems to be compatible with both SELinux and the Google Play policy.

@TheBrokenRail

This comment has been minimized.

Copy link

@TheBrokenRail TheBrokenRail commented Nov 19, 2019

Why not fork() Termux and then load proot as a shared library, this way even if the app library folder becomes noexec, it should still work? (Unless fork() is blocked.)

@bndeff

This comment has been minimized.

Copy link

@bndeff bndeff commented Nov 19, 2019

As far as I understand the issue is not with loading proot (which we can bundle with termux) but with running dynamic code downloaded via apt, pip, gem, cpan etc. or compiled from source. Running dynamic code makes it impossible to audit apps for malware, so its use is disallowed via Google Play policy and increasingly hardening SELinux rulesets. As @thestinger noted earlier, there are still a few loopholes in SELinux in Q, but on the one hand they are still against Google Play policies, on the other hand they will eventually be closed and then we are back to square one. But there is one legal way to run untrusted code: put it in a restricted sandbox by starting a service with the isolatedProcess property set in the manifest. (We still need some extra code – perhaps based on proot – to set up the environment in this sandbox and communicate with it.)

There was a discussion on gitter starting at https://gitter.im/termux/dev?at=5dbfc4cd3d669b28a03014f2 mostly involving @fornwall, @xeffyr and @RalfWerner. They started with 3 possible options and mostly converged on using apks or dynamic feature modules for package installation. They are also trying to find some auxiliary method (qemu, proot or whatever) for user-compiled code. I was missing the mention of isolatedProcess / isolated_app from this chat, that's why I was asking if it has been previously excluded (somewhere in the long scrollback in the chat) or it was just simply forgotten (because of the long scrollback here).

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Nov 19, 2019

Usage isolatedProcess will make things a lot more complicated. If I understand correctly (from docs: this service will run under a special process that is isolated from the rest of the system and has no permissions of its own. The only communication with it is through the Service API), things that will run under such "isolation" will not have any permissions, so any networking & storage access will be disabled. Perhaps isolated process even running with different user id (if so, there will be problems with accessing files due to ownership issues).

@RalfWerner

This comment has been minimized.

Copy link

@RalfWerner RalfWerner commented Nov 19, 2019

@bndeff I entered the Q-Discussion here to use my development in higher Android versions and saved the example with full environment. You wrote above:

  • We still need some extra code – perhaps based on proot – to set up the environment in this sandbox and communicate with it

Could you describe your idea there with practical steps?
Leonid is skeptical as to whether this solution has any advantages. I miss the experience with all suggestions and would test them as far as possible.

@TheBrokenRail

This comment has been minimized.

Copy link

@TheBrokenRail TheBrokenRail commented Nov 19, 2019

@xeffyr

This comment has been minimized.

Copy link
Member

@xeffyr xeffyr commented Nov 19, 2019

Maybe, I didn't checked its code.

However, utility ps showing that sandboxed processes have random user ids:

u:r:isolated_app:s0:c512,c768  u0_i127       3735  5512 1757388 147852 SyS_epoll_wait      0 S com.android.chrome:sandboxed
u:r:isolated_app:s0:c512,c768  u0_i128       3764  5512 1836464 181892 SyS_epoll_wait      0 S com.android.chrome:sandboxed
u:r:untrusted_app:s0:c512,c768 u0_a91       29580  4727 1948400 199000 SyS_epoll_wait      0 S com.android.chrome
u:r:isolated_app:s0:c512,c768  u0_i124      29793  5512 1800012  48016 SyS_epoll_wait      0 S com.android.chrome:sandboxed
u:r:untrusted_app:s0:c512,c768 u0_a91       29840  4727 1870776 106160 SyS_epoll_wait      0 S com.android.chrome:privileged_process0
u:r:isolated_app:s0:c512,c768  u0_i125      29913  5512 1734300  38952 SyS_epoll_wait      0 S com.android.chrome:sandboxed

And there really may be problems with file system access and networking:

herolte:/ $ cat /proc/29913/status
Name:	dboxed_process1
State:	S (sleeping)
Tgid:	29913
Ngid:	0
Pid:	29913
PPid:	5512
TracerPid:	0
Uid:	99125	99125	99125	99125
Gid:	99125	99125	99125	99125
FDSize:	128
Groups:	

- supplementary groups are empty.

@thestinger

This comment has been minimized.

Copy link

@thestinger thestinger commented Nov 20, 2019

You can't have internet access, etc. directly from there. It has to be proxied to a broker process outside of it.

@michalbednarski

This comment has been minimized.

Copy link
Contributor

@michalbednarski michalbednarski commented Nov 20, 2019

I've done few tests with isolatedProcess and:

  • socket(AF_INET) cannot be used there (socket() fails with EACCES)
  • socket(AF_INET) file descriptor cannot be passed there from app through Binder
    • I haven't tested unix socket but I suspect it'll behave same
  • isolatedProcess cannot map files from /data/data as PROT_READ|PROT_EXEC
    • it can map them as PROT_READ
    • they currently can be mapped inside untrusted_app (that is, where Termux currently runs)
      • This probably will go away in future, current implementation bans execve() from data but still allows dlopen()ing (System.load in Java) downloaded files
      • This is why using proot with targetSdkVersion 29 still works while direct execve() doesn't
      • I'd prefer building thin termux-exec based wrapper with just ELF loader instead of using full proot all the time
        • Maybe this is just me having too low perception of stability of proot
        • I'm pretty sure proot nests badly
        • Doing so is technically still against Google Play policy (that is proot and custom ELF loader running in untrusted_app are unsandboxed downloaded code)
  • What Chromium does is mapping anonymous executable memory
    • That currently works both in app and isolated process
    • Presumably in future it will be possible only from isolated process
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.