mitmproxy does not see traffic from Android app

[bridgewaterrobbie]

Steps to reproduce the problem:

1. Install mitmproxy's CA certificate on Android.
2. Use Chrome to verify that HTTPS interception is working.
3. Open an app to inspect its HTTPS traffic.

Expected result

App connects via proxy and traffic is shown in mitmproxy.

Actual result

App rejects mitmproxy's certificate and fails to connect to server.

Any other comments? What have you tried so far?

The behavior seems similar to what happens when you try to connect to an https site before installing the certificate - no network traffic is seen until you accept the insecure connection. The same app can be intercepted on iOS and Android 6.

System information

Google Pixel running Android 7.1.1

[mhils]

I'm afraid this is the first bug report where we see the changes to trusted Certificate Authorities in Android Nougat in action. In a nutshell, apps on Android Nougat only accept certs from the system CA store, user-added CAs are not considered unless the application explicitly opts in.

I tried to ask Google folks about this at a couple of places (e.g. here). Sorry for the drastic words, but my preliminary conclusion is that the Android team just doesn't give a shit about your privacy. My only guess for their motivation is that they want to avoid bad press for apps with privacy violations? I just don't see the threat model they are addressing with this change. It's a bit ironic that your device considers the Web PKI as more trustworthy than you.

It's a bit of a shame for Google as the Chrome security team does a fantastic job at advancing the state of the TLS ecosystem, but they are apparently not talking to the Android team. ¯\_(ツ)_/¯ If you know any Google folks involved in that decision, we'd be very happy to talk and hear their point of view.

[victorhooi]

There was a discussion about this earlier on HN:

https://news.ycombinator.com/item?id=12061320

I hit this issue myself this morning, trying to debug an issue in an Android app as well.

The mitmproxy docs (http://docs.mitmproxy.org/en/stable/certinstall.html) mention this in passing:

If you want to intercept the pinned connections, you need to patch the application manually. For Android and (jailbroken) iOS devices, various tools exist to accomplish this.

But they don't actually mention what these various tools are.

I believe if you have a rooted Android device, and XPosed installed - perhaps you could use:

https://github.com/Fuzion24/JustTrustMe

However, I haven't verified it myself. If somebody is able to verify it - should we reference it in the docs?

Any other approaches people can suggest?

[victorhooi]

Hmm Xposed doesn't support Android 7.x =(.

Any other suggestions?

[bridgewaterrobbie]

I added the "Trusting custom CAs for debugging" code (With a modification since the XML is wrong), but it did not allow the app to go through the proxy.

Might be notable though, I could use and track a button that spawns a web view, just not the login functionality. This makes me think the issue for me isn't entirely google, it might be partly cognito(Again, my iOS version works fine).

I also think it aught to work if you install the mitm certificate as a system trusted certificate, but am unsure how to go about it (It should be possible when rooted though right?)

[mhils]

@chadbrubaker, I see you authored the Android blog post, can you shed some light on the team's motivation for this change in the trust policy? Sorry for the cold mention, just curious what's the reasoning behind all this.

(For the record, the new API looks considerably better, but the hiding-traffic-from-users aspect is very irritating)

[chadbrubaker]

The crux of the motivation is in the blog post -- application data is secure by default. Our goal is to protect end users, so while I enjoyed the snarky analysis up thread that's not at all why :). Protecting application data is protecting user data and by extension user privacy at a fundamental level.

The change does add additional work for reverse engineering or testing applications, and I'm sorry for that, but neither of those tasks are in the threat model for this feature. It does not prevent nor attempt to prevent you from doing those kinds of things.

It always been the case that an application could choose not to trust your MiTM CA and there are many things you can do as a reverse engineer in those situations. Patching the TLS code in OS is almost always massive overkill and solving the problem at the wrong level, IMO, but how you want to go about that is up to you.

If you do find apps with security or privacy issues do let us know at security@android.com.

@victorhooi Keep in mind that XPosed module disables all TLS verification which is something you should never do and makes it trivially MiTMable, don't use that on a device with private data and wipe the device when you're done.

@bridgewaterrobbie debug-overrides are only used when android:debuggable is set. The config under 'Trusting user-added CAs for all secure connections' should work just fine for you. Also sorry about that typo in the blog, oops!

[mhils]

@chadbrubaker: Thanks for getting back so quickly - I appreciate it!

Our goal is to protect end users, so while I enjoyed the snarky analysis up thread that's not at all why :). Protecting application data is protecting user data and by extension user privacy at a fundamental level.

What is your threat model regarding user-added CAs? In all other settings I know of, user-added CAs are considered particularly trustworthy. For example, Chrome, Firefox and IE will not enforce HPKP if they encounter a cert from a user-added CA. Why does Android consider user-added CAs less trustworthy than normal CAs, whereas all other major platforms do the contrary?

The change does add additional work for reverse engineering or testing applications, and I'm sorry for that, but neither of those tasks are in the threat model for this feature. It does not prevent nor attempt to prevent you from doing those kinds of things.

This may not be your intention, but unfortunately I feel it's a practical consequence. You and I know how to circumvent that, but journalists at the WSJ, information privacy lawyers or the guy who found that Path uploaded your address book probably won't bother as much. A lot of those issues are just found by people who don't run custom roms. Not providing a developer option for this raises the entry bar by a lot.

Sorry if this comes off harsh - I really don't want to argue against more secure defaults, but I'd appreciate a good explanation what leads your team think that Android should behave different than all other ecosystems here. If there are good reasons, I'd love to hear them here or privately if you'd prefer to take it off-list (email is on my GitHub).

Thanks!
Max

[bridgewaterrobbie]

Just throwing out, after using

<network-security-config>  
     <debug-overrides>  
          <trust-anchors>  
               <!-- Trust preinstalled CAs -->  
               <certificates src="system" />  
               <!-- Additionally trust user added CAs -->  
               <certificates src="user" />  
          </trust-anchors>  
     </debug-overrides>  
</network-security-config>

in my network security config things started working. Not sure if the system part was needed, but wanted to cover everything so I could get back to work.

[cortesi]

@chadbrubaker Any further comment on this issue? I think everyone can agree that the practical impact of this move will be to greatly reduce the amount of scrutiny that apps receive. I strongly feel that the net effect will be negative for users. We're nowhere near the point where the Android ecosystem is well-controlled enough to tilt the balance away from transparency in this way.

[javabrett]

Would the option to trust user certificates be a good candidate for adding to Developer Mode preferences? Assuming that dialog is well secured, this should strike a good balance between user privacy and security developer need.

It would be a great shame to lose this ability on Android, or for it to require custom roms.

[victorhooi]

Has anybody looked into this, and whether it works?

https://github.com/levyitay/AddSecurityExceptionAndroid

Apparently you can simply pass this an APK file, and it will automate adding the network exceptions into it (i.e. the stuff above in #2054 (comment)) It also requires apktool:

https://ibotpeaches.github.io/Apktool/

[raymontag]

@victorhooi Thanks worked like a charm for several apps.

@mhils @cortesi Maybe you should consider to put this as a workaround into the official documentation. My guess is that the situation will not change anytime soon.

[bogomazov]

I don't like the idea of recompiling different apks, is there any update in here? I would really appreciate a universal solution for all apps installed on my device.

[Lekensteyn]

Ran into this issue as well while trying to check for privacy implications of an app. The blog post and docs linked above does explain what happened, but not how. Source code references follows based on nougat-mr2.3.

The change was made in commit aosp-mirror/platform_frameworks_base@32d2a10, see frameworks/base:core/java/android/security/net/config/NetworkSecurityConfig.java. After that change, only the system root store (/etc/security/cacerts) is considered.

NetworkSecurityConfig.getDefaultBuilder is used in XmlConfigSource.java when parsing the configuration file, if the option is not found in the APK, then it uses that default (which only includes the system root).

XmlConfigSource (and the fallback to getDefaultBuilder) is used in NetworkSecurityConfig, which is used all over the place. And unlike permissions, this configuration does not seem cached in /data/system/.

---

Editing each APK after installation and upgrades seem fragile...

So let's turn to the alternative since I have root, adding the certificate to the root store in /etc/security/cacerts/. Certificates here are stored based on the hash of the Subject field plus an index number (increasing from 0; in case the same certificate is issues with multiple hash algorithms, e.g. SHA-1 and SHA-256).

$ openssl x509 -noout -subject_hash_old -in ~/.mitmproxy/mitmproxy-ca-cert.pem
c8750f0d

Now you need to copy your PEM-encoded CA file to the system partition, like:

$ adb root
$ adb push ~/.mitmproxy/mitmproxy-ca-cert.pem /etc/security/cacerts/c8750f0d.0
$ adb shell chmod 644 /etc/security/cacerts/c8750f0d.0

In my case, my /system was readonly and I could not write it. So as workaround, I cloned the cacerts directory from system to userdata, added the cert and bind-mounted it over the system directory.

ca=~/.mitmproxy/mitmproxy-ca-cert.pem
hash=$(openssl x509 -noout -subject_hash_old -in $ca)
adb root
adb shell cp -a /etc/security/cacerts /data/
adb push $ca /data/cacerts/$hash.0
adb shell chmod 644 /data/cacerts/$hash.0
adb shell mount --bind /data/cacerts /etc/security/cacerts
adb unroot  # optional: restart adbd as normal user

Restarted the app, and it works! No more "Trust anchor for certification path not found." errors!

Note: this needs to be done once every time you restart the device and want to perform a mitm.