mitmproxy does not see traffic from Android app

[bridgewaterrobbie]

Steps to reproduce the problem:

1. Install mitmproxy's CA certificate on Android.
2. Use Chrome to verify that HTTPS interception is working.
3. Open an app to inspect its HTTPS traffic.

Expected result

App connects via proxy and traffic is shown in mitmproxy.

Actual result

App rejects mitmproxy's certificate and fails to connect to server.

Any other comments? What have you tried so far?

The behavior seems similar to what happens when you try to connect to an https site before installing the certificate - no network traffic is seen until you accept the insecure connection. The same app can be intercepted on iOS and Android 6.

System information

Google Pixel running Android 7.1.1

[mhils]

I'm afraid this is the first bug report where we see the changes to trusted Certificate Authorities in Android Nougat in action. In a nutshell, apps on Android Nougat only accept certs from the system CA store, user-added CAs are not considered unless the application explicitly opts in.

I tried to ask Google folks about this at a couple of places (e.g. here). Sorry for the drastic words, but my preliminary conclusion is that the Android team just doesn't give a shit about your privacy. My only guess for their motivation is that they want to avoid bad press for apps with privacy violations? I just don't see the threat model they are addressing with this change. It's a bit ironic that your device considers the Web PKI as more trustworthy than you.

It's a bit of a shame for Google as the Chrome security team does a fantastic job at advancing the state of the TLS ecosystem, but they are apparently not talking to the Android team. ¯\_(ツ)_/¯ If you know any Google folks involved in that decision, we'd be very happy to talk and hear their point of view.

[victorhooi]

There was a discussion about this earlier on HN:

https://news.ycombinator.com/item?id=12061320

I hit this issue myself this morning, trying to debug an issue in an Android app as well.

The mitmproxy docs (http://docs.mitmproxy.org/en/stable/certinstall.html) mention this in passing:

If you want to intercept the pinned connections, you need to patch the application manually. For Android and (jailbroken) iOS devices, various tools exist to accomplish this.

But they don't actually mention what these various tools are.

I believe if you have a rooted Android device, and XPosed installed - perhaps you could use:

https://github.com/Fuzion24/JustTrustMe

However, I haven't verified it myself. If somebody is able to verify it - should we reference it in the docs?

Any other approaches people can suggest?

[victorhooi]

Hmm Xposed doesn't support Android 7.x =(.

Any other suggestions?

[bridgewaterrobbie]

I added the "Trusting custom CAs for debugging" code (With a modification since the XML is wrong), but it did not allow the app to go through the proxy.

Might be notable though, I could use and track a button that spawns a web view, just not the login functionality. This makes me think the issue for me isn't entirely google, it might be partly cognito(Again, my iOS version works fine).

I also think it aught to work if you install the mitm certificate as a system trusted certificate, but am unsure how to go about it (It should be possible when rooted though right?)

[mhils]

@chadbrubaker, I see you authored the Android blog post, can you shed some light on the team's motivation for this change in the trust policy? Sorry for the cold mention, just curious what's the reasoning behind all this.

(For the record, the new API looks considerably better, but the hiding-traffic-from-users aspect is very irritating)

[chadbrubaker]

The crux of the motivation is in the blog post -- application data is secure by default. Our goal is to protect end users, so while I enjoyed the snarky analysis up thread that's not at all why :). Protecting application data is protecting user data and by extension user privacy at a fundamental level.

The change does add additional work for reverse engineering or testing applications, and I'm sorry for that, but neither of those tasks are in the threat model for this feature. It does not prevent nor attempt to prevent you from doing those kinds of things.

It always been the case that an application could choose not to trust your MiTM CA and there are many things you can do as a reverse engineer in those situations. Patching the TLS code in OS is almost always massive overkill and solving the problem at the wrong level, IMO, but how you want to go about that is up to you.

If you do find apps with security or privacy issues do let us know at security@android.com.

@victorhooi Keep in mind that XPosed module disables all TLS verification which is something you should never do and makes it trivially MiTMable, don't use that on a device with private data and wipe the device when you're done.

@bridgewaterrobbie debug-overrides are only used when android:debuggable is set. The config under 'Trusting user-added CAs for all secure connections' should work just fine for you. Also sorry about that typo in the blog, oops!

[mhils]

@chadbrubaker: Thanks for getting back so quickly - I appreciate it!

Our goal is to protect end users, so while I enjoyed the snarky analysis up thread that's not at all why :). Protecting application data is protecting user data and by extension user privacy at a fundamental level.

What is your threat model regarding user-added CAs? In all other settings I know of, user-added CAs are considered particularly trustworthy. For example, Chrome, Firefox and IE will not enforce HPKP if they encounter a cert from a user-added CA. Why does Android consider user-added CAs less trustworthy than normal CAs, whereas all other major platforms do the contrary?

The change does add additional work for reverse engineering or testing applications, and I'm sorry for that, but neither of those tasks are in the threat model for this feature. It does not prevent nor attempt to prevent you from doing those kinds of things.

This may not be your intention, but unfortunately I feel it's a practical consequence. You and I know how to circumvent that, but journalists at the WSJ, information privacy lawyers or the guy who found that Path uploaded your address book probably won't bother as much. A lot of those issues are just found by people who don't run custom roms. Not providing a developer option for this raises the entry bar by a lot.

Sorry if this comes off harsh - I really don't want to argue against more secure defaults, but I'd appreciate a good explanation what leads your team think that Android should behave different than all other ecosystems here. If there are good reasons, I'd love to hear them here or privately if you'd prefer to take it off-list (email is on my GitHub).

Thanks!
Max

[bridgewaterrobbie]

Just throwing out, after using

<network-security-config>  
     <debug-overrides>  
          <trust-anchors>  
               <!-- Trust preinstalled CAs -->  
               <certificates src="system" />  
               <!-- Additionally trust user added CAs -->  
               <certificates src="user" />  
          </trust-anchors>  
     </debug-overrides>  
</network-security-config>

in my network security config things started working. Not sure if the system part was needed, but wanted to cover everything so I could get back to work.

[cortesi]

@chadbrubaker Any further comment on this issue? I think everyone can agree that the practical impact of this move will be to greatly reduce the amount of scrutiny that apps receive. I strongly feel that the net effect will be negative for users. We're nowhere near the point where the Android ecosystem is well-controlled enough to tilt the balance away from transparency in this way.

[javabrett]

Would the option to trust user certificates be a good candidate for adding to Developer Mode preferences? Assuming that dialog is well secured, this should strike a good balance between user privacy and security developer need.

It would be a great shame to lose this ability on Android, or for it to require custom roms.

[victorhooi]

Has anybody looked into this, and whether it works?

https://github.com/levyitay/AddSecurityExceptionAndroid

Apparently you can simply pass this an APK file, and it will automate adding the network exceptions into it (i.e. the stuff above in #2054 (comment)) It also requires apktool:

https://ibotpeaches.github.io/Apktool/

[raymontag]

@victorhooi Thanks worked like a charm for several apps.

@mhils @cortesi Maybe you should consider to put this as a workaround into the official documentation. My guess is that the situation will not change anytime soon.

[bogomazov]

I don't like the idea of recompiling different apks, is there any update in here? I would really appreciate a universal solution for all apps installed on my device.

[Lekensteyn]

Ran into this issue as well while trying to check for privacy implications of an app. The blog post and docs linked above does explain what happened, but not how. Source code references follows based on nougat-mr2.3.

The change was made in commit aosp-mirror/platform_frameworks_base@32d2a10, see frameworks/base:core/java/android/security/net/config/NetworkSecurityConfig.java. After that change, only the system root store (/etc/security/cacerts) is considered.

NetworkSecurityConfig.getDefaultBuilder is used in XmlConfigSource.java when parsing the configuration file, if the option is not found in the APK, then it uses that default (which only includes the system root).

XmlConfigSource (and the fallback to getDefaultBuilder) is used in NetworkSecurityConfig, which is used all over the place. And unlike permissions, this configuration does not seem cached in /data/system/.

---

Editing each APK after installation and upgrades seem fragile...

So let's turn to the alternative since I have root, adding the certificate to the root store in /etc/security/cacerts/. Certificates here are stored based on the hash of the Subject field plus an index number (increasing from 0; in case the same certificate is issues with multiple hash algorithms, e.g. SHA-1 and SHA-256).

$ openssl x509 -noout -subject_hash_old -in ~/.mitmproxy/mitmproxy-ca-cert.pem
c8750f0d

Now you need to copy your PEM-encoded CA file to the system partition, like:

$ adb root
$ adb push ~/.mitmproxy/mitmproxy-ca-cert.pem /etc/security/cacerts/c8750f0d.0
$ adb shell chmod 644 /etc/security/cacerts/c8750f0d.0

In my case, my /system was readonly and I could not write it. So as workaround, I cloned the cacerts directory from system to userdata, added the cert and bind-mounted it over the system directory.

ca=~/.mitmproxy/mitmproxy-ca-cert.pem
hash=$(openssl x509 -noout -subject_hash_old -in $ca)
adb root
adb shell cp -a /etc/security/cacerts /data/
adb push $ca /data/cacerts/$hash.0
adb shell chmod 644 /data/cacerts/$hash.0
adb shell mount --bind /data/cacerts /etc/security/cacerts
adb unroot  # optional: restart adbd as normal user

Restarted the app, and it works! No more "Trust anchor for certification path not found." errors!

Note: this needs to be done once every time you restart the device and want to perform a mitm.

[ilpianista]

@Lekensteyn thank you for the workaround!
However, there's no need to bind /data/cacerts, just execute
adb shell mount -o remount,rw /system
copy the certificate and remount again with
adb shell mount -o remount,ro /system

[Lekensteyn]

@ilpianista Unfortunately that does not work on all systems. In my case I was using Android-X86 where /system is a ro CD image. Another case where it could (presumably) have bad side-effects is when "verified boot" is enabled, simply mounting a filesystem would change the filesystem metadata and fail the verification.

[ilpianista]

@Lekensteyn ah, I see. Thank you a lot for explaining.

[victorhooi]

If you have root access, this blog post might help:

https://blog.jeroenhd.nl/article/android-7-nougat-and-certificate-authorities

[fieryo]

It seems that this research lab has released a module that is able to install client certificates on Android Nougat. There is even a screenshot with mitmproxy cert installed. Module is explained as: Universal Systemless Interface, to create an altered mask of the system without changing the system itself.

Did someone test this solution or it also requires a rooted device (or recompiled apps)? Removing client cert after testing is essential as well.

[Lekensteyn]

@fiery- That module seems to use Magisk which is used to root phones. Assuming it does not write directly to /system, I guess that it works using bind mounts as described in #2054 (comment)

[fieryo]

@Lekensteyn Indeed, they use Magisk to bind-mount a user certs folder to the /system. However, installing Magisk seems to be a task for very advanced users.

Magisk's developer posted on Twitter that there is a fake site and a fake MagiskManager app on Google Play Store:

The official Github repository hosts only source and is not helpful how to install/uninstall this module (on non rooted devices). Some devices (i.e. Samsung) detect when you root them, so the risk to get it wrong is not small.

[hanassi]

@Lekensteyn
I've try to place "burp suite CA" at /etc/security/cacerts at my device, and I made it system certificate. but proxy still not working. In your way, It does not work in "burp suite CA"?

[Lekensteyn]

@hanassi Have you followed the exact steps as I described above, including making sure that the fingerprint is used as filename? What does "not working" mean? If you connect to https://example.com while your MITM is active, do you see your CA?

Have you looked with Wireshark what traffic (and certificate) is exchanged?

[hanassi]

@Lekensteyn yes, I download "burp suite CA" and rename certificate by use openssl command.

openssl x509 -inform der -outform pem -out cacert.crt -in cacert.der
openssl x509 -noout -subject_hash_old -in cacert.crt

[pohmelie]

I also have problems with android nougat. I made everything as @Lekensteyn said, except that I can remount /system, so I see mitm proxy in list of trusted system credentials. But I'm still do not see some apps traffic through mitm proxy. In wireshark I see some handshaking and

1590 10.188781010 192.168.1.33 192.168.1.42 TLSv1.2 73 Alert (Level: Fatal, Description: Unknown CA)

And tcp rst after.

Do not really now, maybe default mitm certificate is blacklisted or some anti-mitm technics in apps twitter, whatsapp, instagram, dropbox, youtube, uber are done. But reddit, google maps, google mail works just fine and I see all requests.

[wych42]

@Lekensteyn I got this error after mount --bind and open APP: "trust anchor for certification path not found"

[Jonahss]

@Lekensteyn I followed your instructions, but for some reason, my emulator claims it doesn't have permission to read the new certificate file.
I've checked the permissions:

~> adb shell ls -la /etc/security/cacerts | grep c8750f0d
-rw-r--r-- 1 root root 1318 2019-03-20 11:29 c8750f0d.0

And here's logcat complaining about not being able to read the certificate file:

03-20 12:33:52.678  4641  5106 E DirectoryCertificateSrc: Failed to read certificate from c8750f0d.0
03-20 12:33:52.678  4641  5106 E DirectoryCertificateSrc: java.io.FileNotFoundException: /system/etc/security/cacerts/c8750f0d.0 (Permission denied)
03-20 12:33:52.678  4641  5106 E DirectoryCertificateSrc:       at java.io.FileInputStream.open0(Native Method)

Any idea what else I can do?

[Jonahss]

Answer to my own question^

I noticed that not only was my new cert not readable, none of the default system certs were either. They all disappeared from the list in Settings.

What I needed to do for my emulator was adb remount. But this was blocked by all the methods I saw online except for starting the emulator with the -writeable-system flag. Now I'm mitm'ing :D

[Keshvadi]

I read all of the comments, but what is the conclusion. Is there any way to intercept apps traffic?

[Jonahss]

@Keshvadi totally possible. I wrote a blog post about it: https://appiumpro.com/editions/63

It contains instructions on how to set up the emulators. You can run mitmproxy from commandline rather than the library I write about and you can ignore Appium stuff if you're not testing.

All I did was use lekenstyen's answer, but had to add a flag when starting the emulator.
#2054 (comment)

[KaKi87]

Hello,

This tutorial aso works : install certificates the classic way then move the files from /data/misc/user/0/cacerts-added to /system/etc/security/cacerts.

++

[RoseDeSable]

That's correct. But I have a Samsung Tablet with version 8.1.0 of android. Here I it is impossible to mount the system partition. Rooting the tablet is not possible or too risky.

Therefore it is necessary to say, that the phone's security concept can be very good: The point of cracking the traffic are the apps itsself. It is interresting to see, that it is possible to intercept some apps with a root-certificate in the local user storage. It is dependent on the manner, how the apps are wrote. The traffic of apps like "aem" or "bmw connected drive" can be intercepted, but apps like "katwarn", "youtube" aren't breakable.

Some apps are using the browser of the tablet for login. Because the browser is interceptable with a certificate in the user storage, you can capture the user's password. Therefore I don't understand, why the programmers of the cloud apps - nextcloud or owncloud - use this insecure way for authentication.

On the other side after md5 the algorithm sha1 is also weak because of the increasing cpu and/or gpu power: collisions are quicker found as the years ago. So someone could copy the signature of a server certificate, that is signed by an authourized instance in the internet, to its own self signed root-certificate: This is possible in the case, where the hash of its own certificate is equal to the hash of the server certificate (collosion). I believe, that in this case the interception of nearly whole the traffic of a phone is possible with mitmproxy. The exception ar apps, that use certificate pinning.

[Junyan]

It's working for me.

• Install VirtualXposed APP and JustTrustMe module.
• Install your APP in VirtualXposed.
• Open your APP in VirtualXposed.

references:

• https://github.com/android-hacker/VirtualXposed
• https://github.com/Fuzion24/JustTrustMe

[gah-bo]

It's working for me.

I'm trying to use VirtualXposed APP and JustTrustMe so that I can do this with WeedMaps to extract menu data from my dispensary, inspired by https://www.youtube.com/watch?v=Fa1zlLEGPtw .

mitimproxy keeps disconnecting/connecting on me and apps like WM just wont work. However i tried this VirtualXposed APP + JustTrustMe combo and when I open WM it gives me a prompt to update play services. I tried this https://repo.xposed.info/module/com.aviraxp.nomoregoogleplayservices to no avail. I cannot enter WM on VirtualXposed APP because that prompt stops me.

I am not rooted btw.

[RoseDeSable]

Hello,
I find this page https://virtualxposed.com/ in the internet. If I downloaded the newest version of the app from the page, I see the reference to github: https://github.com/android-hacker/VirtualXposed/releases/download/0.18.2/VirtualXposed_0.18.2.apk. Exist a reason to build the app from source (https://github.com/android-hacker/VirtualXposed) with apktool, or can I use the downloaded apk for testing ?

Bye
Rose

[Mattwmaster58]

I almost had mitmproxy working for Android, but the cert generated by mitmproxy expired 2.5y in the future, which some random guy on the internet thinks means Android disregards it. In my own testing this is certainly the case.

Here's what I did to almost make things work:

• retrieved the cert.pem from mitm.it
• followed the steps from the aforementioned SO answer to place the cert in the appropriate location (I used my recovery to place the file and chmod 644)
• connected to the proxy through ProxyDroid (requires root)

[iuriatan]

I just would like to comment that this tutorial on additional configuration steps for andoid N and above helped me opt in the mitm proxy for the app I'm working on. A simple change in the app manifest can override the nougat restrictions. I strongly believe it deserves mentioning in mitmproxy docs!

[kpj]

In case someone is looking for a solution which does not require modifying the APK's manifest, the trick which made everything work for me is to:

• use API level 28 with Google APIs (not Google Play)
• start the emulator with -writable-system
• run adb root && adb remount

This is basically a combination of various previous posts in this thread (and is probably also not future-proof), but should allow you to install the required system certificates without issues.
I did a slightly more detailed write-up here.

Interception Example

Hope this is of help to some :)

[sinhpn92]

Have any suggest to sniff traffic from Android app on Android 8+?

[RoseDeSable]

For sniffing the traffic of a specific app the app "SandroProxy" is a first good solution. In the column "APPS" you see all the traffic seperated for the single apps. You can install a self-signed root certificate for intercepting ssl-traffic. But also here: You must test, for which app you can do the interception. Apps like Youtube only accept certificate of the system storage. With an outside proxy like mitmproxy you are not able to exactly separate the traffic for a specific app !

[thijs-qv]

I was able to use mitmproxy to inspect app traffic on an emulator running Android 29. I followed the steps of @KaKi87, but had to do some extra things to make /system writable.

• create an emulator with Google API (not Google Play).
• start the emulator with the -writable-system option
• enable the proxy by setting it in the emulator settings via Settings > Proxy
• install the mitm certificate (via mitm.it) on the emulator
• Now run the following:
  adb root
adb shell avbctl disable-verification
adb reboot
• When the emulator has rebooted we can make the /system directory writable:
  adb root
adb remount
• /system is now writable, so copy the user certificate from /data/misc/user/0/cacerts-added to /system/etc/security/cacerts

[aptarmy]

If you don't want to root your physical mobile device like me, you can add Network Security Configuration to the apk file as mentioned by @victorhooi
here, but it only works with Android below 11. For Android 11, If you use AddSecurityExceptionAndroid you won't be able to install the modified apk, as your device won't accept older version of apk signing scheme.

So for Android 11, you can use somthing else like apk-mitm. It works great with my mobile device.

install it using
npm install -g apk-mitm

[AtitBimali]

hey @aptarmy I tried few apps with apk-mitm, it successfully rebuilds the apk and the apk also gets install on my phone without any issue. But when i launch the app, it closes immediately.

[snimavat]

On Android 13 emulator : configured to use proxy (from android studio)

Android App or chrome traffic just does not go through the proxy, it behaves like there's no proxy)
but android studio traffic does go through proxy and i can see it in logs.

Any idea, it feels like avd is just ignoring the proxy settings.

[s1nistr4]

This is still an issue in 2023, with 55+ likes on the issue report. Please fix this issue, it should be top priority. This is the only thing keeping a lot of other users and I from using mitmproxy for everything.

[3052]

I think this issue should be closed. I support the basic premise, but the original issue as posted is too vague to be actionable. we need to know WHAT app is having trouble, as the solution is highly dependent on what if any protections the app is using, some of which might be out of scope for MITM Proxy.

[KaKi87]

I think this issue should be closed.

Agreed.

the original issue as posted is too vague to be actionable. we need to know WHAT app is having trouble, as the solution is highly dependent on what if any protections the app is using, some of which might be out of scope for MITM Proxy.

Not for that reason, though.

On the contrary, the issue isn't app-specific and was clearly idenfied as the SSL certificate trust issue, for which several people (including myself) shared a fix, despite requiring root. Case solved.

In the meantime, I'm still waiting on #4408 and #5432.