New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mitmproxy does not see traffic from Android app #2054

Open
bridgewaterrobbie opened this Issue Feb 22, 2017 · 29 comments

Comments

Projects
None yet
@bridgewaterrobbie
Copy link

bridgewaterrobbie commented Feb 22, 2017

Steps to reproduce the problem:
  1. Install mitmproxy's CA certificate on Android.
  2. Use Chrome to verify that HTTPS interception is working.
  3. Open an app to inspect its HTTPS traffic.
Expected result

App connects via proxy and traffic is shown in mitmproxy.

Actual result

App rejects mitmproxy's certificate and fails to connect to server.

Any other comments? What have you tried so far?

The behavior seems similar to what happens when you try to connect to an https site before installing the certificate - no network traffic is seen until you accept the insecure connection. The same app can be intercepted on iOS and Android 6.

System information

Google Pixel running Android 7.1.1

@mhils

This comment has been minimized.

Copy link
Member

mhils commented Feb 22, 2017

I'm afraid this is the first bug report where we see the changes to trusted Certificate Authorities in Android Nougat in action. In a nutshell, apps on Android Nougat only accept certs from the system CA store, user-added CAs are not considered unless the application explicitly opts in.

I tried to ask Google folks about this at a couple of places (e.g. here). Sorry for the drastic words, but my preliminary conclusion is that the Android team just doesn't give a shit about your privacy. My only guess for their motivation is that they want to avoid bad press for apps with privacy violations? I just don't see the threat model they are addressing with this change. It's a bit ironic that your device considers the Web PKI as more trustworthy than you.

It's a bit of a shame for Google as the Chrome security team does a fantastic job at advancing the state of the TLS ecosystem, but they are apparently not talking to the Android team. ¯\_(ツ)_/¯ If you know any Google folks involved in that decision, we'd be very happy to talk and hear their point of view.

@mhils mhils added the upstream label Feb 23, 2017

@victorhooi

This comment has been minimized.

Copy link

victorhooi commented Feb 24, 2017

There was a discussion about this earlier on HN:

https://news.ycombinator.com/item?id=12061320

I hit this issue myself this morning, trying to debug an issue in an Android app as well.

The mitmproxy docs (http://docs.mitmproxy.org/en/stable/certinstall.html) mention this in passing:

If you want to intercept the pinned connections, you need to patch the application manually. For Android and (jailbroken) iOS devices, various tools exist to accomplish this.

But they don't actually mention what these various tools are.

I believe if you have a rooted Android device, and XPosed installed - perhaps you could use:

https://github.com/Fuzion24/JustTrustMe

However, I haven't verified it myself. If somebody is able to verify it - should we reference it in the docs?

Any other approaches people can suggest?

@victorhooi

This comment has been minimized.

Copy link

victorhooi commented Feb 24, 2017

Hmm Xposed doesn't support Android 7.x =(.

Any other suggestions?

@bridgewaterrobbie

This comment has been minimized.

Copy link

bridgewaterrobbie commented Feb 24, 2017

I added the "Trusting custom CAs for debugging" code (With a modification since the XML is wrong), but it did not allow the app to go through the proxy.

Might be notable though, I could use and track a button that spawns a web view, just not the login functionality. This makes me think the issue for me isn't entirely google, it might be partly cognito(Again, my iOS version works fine).

I also think it aught to work if you install the mitm certificate as a system trusted certificate, but am unsure how to go about it (It should be possible when rooted though right?)

@mhils

This comment has been minimized.

Copy link
Member

mhils commented Feb 26, 2017

@chadbrubaker, I see you authored the Android blog post, can you shed some light on the team's motivation for this change in the trust policy? Sorry for the cold mention, just curious what's the reasoning behind all this.

(For the record, the new API looks considerably better, but the hiding-traffic-from-users aspect is very irritating)

@chadbrubaker

This comment has been minimized.

Copy link

chadbrubaker commented Feb 27, 2017

The crux of the motivation is in the blog post -- application data is secure by default. Our goal is to protect end users, so while I enjoyed the snarky analysis up thread that's not at all why :). Protecting application data is protecting user data and by extension user privacy at a fundamental level.

The change does add additional work for reverse engineering or testing applications, and I'm sorry for that, but neither of those tasks are in the threat model for this feature. It does not prevent nor attempt to prevent you from doing those kinds of things.

It always been the case that an application could choose not to trust your MiTM CA and there are many things you can do as a reverse engineer in those situations. Patching the TLS code in OS is almost always massive overkill and solving the problem at the wrong level, IMO, but how you want to go about that is up to you.

If you do find apps with security or privacy issues do let us know at security@android.com.

@victorhooi Keep in mind that XPosed module disables all TLS verification which is something you should never do and makes it trivially MiTMable, don't use that on a device with private data and wipe the device when you're done.

@bridgewaterrobbie debug-overrides are only used when android:debuggable is set. The config under 'Trusting user-added CAs for all secure connections' should work just fine for you. Also sorry about that typo in the blog, oops!

@mhils

This comment has been minimized.

Copy link
Member

mhils commented Feb 27, 2017

@chadbrubaker: Thanks for getting back so quickly - I appreciate it!

Our goal is to protect end users, so while I enjoyed the snarky analysis up thread that's not at all why :). Protecting application data is protecting user data and by extension user privacy at a fundamental level.

What is your threat model regarding user-added CAs? In all other settings I know of, user-added CAs are considered particularly trustworthy. For example, Chrome, Firefox and IE will not enforce HPKP if they encounter a cert from a user-added CA. Why does Android consider user-added CAs less trustworthy than normal CAs, whereas all other major platforms do the contrary?

The change does add additional work for reverse engineering or testing applications, and I'm sorry for that, but neither of those tasks are in the threat model for this feature. It does not prevent nor attempt to prevent you from doing those kinds of things.

This may not be your intention, but unfortunately I feel it's a practical consequence. You and I know how to circumvent that, but journalists at the WSJ, information privacy lawyers or the guy who found that Path uploaded your address book probably won't bother as much. A lot of those issues are just found by people who don't run custom roms. Not providing a developer option for this raises the entry bar by a lot.

Sorry if this comes off harsh - I really don't want to argue against more secure defaults, but I'd appreciate a good explanation what leads your team think that Android should behave different than all other ecosystems here. If there are good reasons, I'd love to hear them here or privately if you'd prefer to take it off-list (email is on my GitHub).

Thanks!
Max

@bridgewaterrobbie

This comment has been minimized.

Copy link

bridgewaterrobbie commented Feb 27, 2017

Just throwing out, after using

<network-security-config>  
     <debug-overrides>  
          <trust-anchors>  
               <!-- Trust preinstalled CAs -->  
               <certificates src="system" />  
               <!-- Additionally trust user added CAs -->  
               <certificates src="user" />  
          </trust-anchors>  
     </debug-overrides>  
</network-security-config>

in my network security config things started working. Not sure if the system part was needed, but wanted to cover everything so I could get back to work.

@cortesi

This comment has been minimized.

Copy link
Member

cortesi commented Mar 13, 2017

@chadbrubaker Any further comment on this issue? I think everyone can agree that the practical impact of this move will be to greatly reduce the amount of scrutiny that apps receive. I strongly feel that the net effect will be negative for users. We're nowhere near the point where the Android ecosystem is well-controlled enough to tilt the balance away from transparency in this way.

@javabrett

This comment has been minimized.

Copy link
Contributor

javabrett commented Mar 16, 2017

Would the option to trust user certificates be a good candidate for adding to Developer Mode preferences? Assuming that dialog is well secured, this should strike a good balance between user privacy and security developer need.

It would be a great shame to lose this ability on Android, or for it to require custom roms.

@victorhooi

This comment has been minimized.

Copy link

victorhooi commented Mar 25, 2017

Has anybody looked into this, and whether it works?

https://github.com/levyitay/AddSecurityExceptionAndroid

Apparently you can simply pass this an APK file, and it will automate adding the network exceptions into it (i.e. the stuff above in #2054 (comment)) It also requires apktool:

https://ibotpeaches.github.io/Apktool/

@raymontag

This comment has been minimized.

Copy link

raymontag commented Jun 22, 2017

@victorhooi Thanks worked like a charm for several apps.

@mhils @cortesi Maybe you should consider to put this as a workaround into the official documentation. My guess is that the situation will not change anytime soon.

@bogomazov

This comment has been minimized.

Copy link

bogomazov commented Aug 18, 2017

I don't like the idea of recompiling different apks, is there any update in here? I would really appreciate a universal solution for all apps installed on my device.

@Lekensteyn

This comment has been minimized.

Copy link

Lekensteyn commented Sep 7, 2017

Ran into this issue as well while trying to check for privacy implications of an app. The blog post and docs linked above does explain what happened, but not how. Source code references follows based on nougat-mr2.3.

The change was made in commit aosp-mirror/platform_frameworks_base@32d2a10, see frameworks/base:core/java/android/security/net/config/NetworkSecurityConfig.java. After that change, only the system root store (/etc/security/cacerts) is considered.

NetworkSecurityConfig.getDefaultBuilder is used in XmlConfigSource.java when parsing the configuration file, if the option is not found in the APK, then it uses that default (which only includes the system root).

XmlConfigSource (and the fallback to getDefaultBuilder) is used in NetworkSecurityConfig, which is used all over the place. And unlike permissions, this configuration does not seem cached in /data/system/.


Editing each APK after installation and upgrades seem fragile...

So let's turn to the alternative since I have root, adding the certificate to the root store in /etc/security/cacerts/. Certificates here are stored based on the hash of the Subject field plus an index number (increasing from 0; in case the same certificate is issues with multiple hash algorithms, e.g. SHA-1 and SHA-256).

$ openssl x509 -noout -subject_hash_old -in ~/.mitmproxy/mitmproxy-ca-cert.pem
c8750f0d

Now you need to copy your PEM-encoded CA file to the system partition, like:

$ adb root
$ adb push ~/.mitmproxy/mitmproxy-ca-cert.pem /etc/security/cacerts/c8750f0d.0
$ adb shell chmod 644 /etc/security/cacerts/c8750f0d.0

In my case, my /system was readonly and I could not write it. So as workaround, I cloned the cacerts directory from system to userdata, added the cert and bind-mounted it over the system directory.

ca=~/.mitmproxy/mitmproxy-ca-cert.pem
hash=$(openssl x509 -noout -subject_hash_old -in $ca)
adb root
adb shell cp -a /etc/security/cacerts /data/
adb push $ca /data/cacerts/$hash.0
adb shell chmod 644 /data/cacerts/$hash.0
adb shell mount --bind /data/cacerts /etc/security/cacerts
adb unroot  # optional: restart adbd as normal user

Restarted the app, and it works! No more "Trust anchor for certification path not found." errors!

Note: this needs to be done once every time you restart the device and want to perform a mitm.

@ilpianista

This comment has been minimized.

Copy link

ilpianista commented Oct 8, 2017

@Lekensteyn thank you for the workaround!
However, there's no need to bind /data/cacerts, just execute
adb shell mount -o remount,rw /system
copy the certificate and remount again with
adb shell mount -o remount,ro /system

@Lekensteyn

This comment has been minimized.

Copy link

Lekensteyn commented Oct 9, 2017

@ilpianista Unfortunately that does not work on all systems. In my case I was using Android-X86 where /system is a ro CD image. Another case where it could (presumably) have bad side-effects is when "verified boot" is enabled, simply mounting a filesystem would change the filesystem metadata and fail the verification.

@ilpianista

This comment has been minimized.

Copy link

ilpianista commented Oct 9, 2017

@Lekensteyn ah, I see. Thank you a lot for explaining.

@victorhooi

This comment has been minimized.

Copy link

victorhooi commented Dec 10, 2017

If you have root access, this blog post might help:

https://blog.jeroenhd.nl/article/android-7-nougat-and-certificate-authorities

@fiery-

This comment has been minimized.

Copy link

fiery- commented Feb 1, 2018

It seems that this research lab has released a module that is able to install client certificates on Android Nougat. There is even a screenshot with mitmproxy cert installed. Module is explained as: Universal Systemless Interface, to create an altered mask of the system without changing the system itself.

Did someone test this solution or it also requires a rooted device (or recompiled apps)? Removing client cert after testing is essential as well.

@Lekensteyn

This comment has been minimized.

Copy link

Lekensteyn commented Feb 2, 2018

@fiery- That module seems to use Magisk which is used to root phones. Assuming it does not write directly to /system, I guess that it works using bind mounts as described in #2054 (comment)

@fiery-

This comment has been minimized.

Copy link

fiery- commented Feb 3, 2018

@Lekensteyn Indeed, they use Magisk to bind-mount a user certs folder to the /system. However, installing Magisk seems to be a task for very advanced users.

Magisk's developer posted on Twitter that there is a fake site and a fake MagiskManager app on Google Play Store:

fake MagiskManager. com site fake MagiskManager app

The official Github repository hosts only source and is not helpful how to install/uninstall this module (on non rooted devices). Some devices (i.e. Samsung) detect when you root them, so the risk to get it wrong is not small.

@hanassi

This comment has been minimized.

Copy link

hanassi commented May 5, 2018

@Lekensteyn
I've try to place "burp suite CA" at /etc/security/cacerts at my device, and I made it system certificate. but proxy still not working. In your way, It does not work in "burp suite CA"?

@Lekensteyn

This comment has been minimized.

Copy link

Lekensteyn commented May 5, 2018

@hanassi Have you followed the exact steps as I described above, including making sure that the fingerprint is used as filename? What does "not working" mean? If you connect to https://example.com while your MITM is active, do you see your CA?

Have you looked with Wireshark what traffic (and certificate) is exchanged?

@hanassi

This comment has been minimized.

Copy link

hanassi commented May 5, 2018

@Lekensteyn yes, I download "burp suite CA" and rename certificate by use openssl command.

openssl x509 -inform der -outform pem -out cacert.crt -in cacert.der
openssl x509 -noout -subject_hash_old -in cacert.crt

1
2
3
4
screenshot_20180506-002244
screenshot_20180506-002403

@pohmelie

This comment has been minimized.

Copy link

pohmelie commented May 8, 2018

I also have problems with android nougat. I made everything as @Lekensteyn said, except that I can remount /system, so I see mitm proxy in list of trusted system credentials. But I'm still do not see some apps traffic through mitm proxy. In wireshark I see some handshaking and

1590 10.188781010 192.168.1.33 192.168.1.42 TLSv1.2 73 Alert (Level: Fatal, Description: Unknown CA)

And tcp rst after.

Do not really now, maybe default mitm certificate is blacklisted or some anti-mitm technics in apps twitter, whatsapp, instagram, dropbox, youtube, uber are done. But reddit, google maps, google mail works just fine and I see all requests.

@BookGin

This comment has been minimized.

Copy link

BookGin commented Sep 8, 2018

Android 7.0

I also have problems with Chrome 64 on my Android 7.0 mobile phone. I have installed the user-added mitmproxy CA, but Chrome cannot connect to any website, even in HTTP. It seems that Chrome refuses to connect if the CA certificate is not system-trusted. Here is the log:

192.168.1.1:46382: clientconnect
192.168.1.1:46382: CONNECT safebrowsing.googleapis.com:443
 << Cannot establish TLS with client (sni: safebrowsing.googleapis.com): TlsException("SSL handshake error: Error([('SSL routines', 'ssl3_read_bytes', 'sslv3 alert certificate unknown')],)",)
192.168.1.1:46382: clientdisconnect

However, to my surprise, Firefox 61.0.2 works very well. I can use Google search without any problem.

Android 8.1

  • Firefox 61.0.2 works, but you have to open the certificate with Firefox and import it manually.
  • Chrome 65.0 works with user-added mitmproxy CA.
  • Both are able to send HTTPS requests without any problem.
@RoseDeSable

This comment has been minimized.

Copy link

RoseDeSable commented Oct 12, 2018

You can intercept all the traffic of your phone also such of the apps. From android 4 upward you must go the way over the sd card.

  1. Insert the sd card in the laptop or other computer

  2. store the certificate mitmproxy-ca-cert.cer of mitmproxy on the sd card

  3. insert the card in the tablet or phone

  4. Now click on the certificate for installing

After this steps you are able to intercept with mitmproxy. It could be, that you additionally must install some root certificates for mobile devices under the cert-directory for mitmproxy

Obviously android up version up seems to accept the sd card as a trustable location, so that you mustn't root the device for installing the certificate under the directory for system certificates. The certificate is stored under user certificate, but android accept it as system certificate !

@mhils

This comment has been minimized.

Copy link
Member

mhils commented Oct 12, 2018

@RoseDeSable: As discussed earlier in the thread this is incorrect.

@RoseDeSable

This comment has been minimized.

Copy link

RoseDeSable commented Oct 15, 2018

Yes, I went wrong, because only the App "AEM" correctly works. It only seems to test the certificat for iwebgis.com in the chain of the certificates. Correctly it is to control the whole chain up to the "Root Certificate" for the instance, that certifies all the followers in the chain, because in android only the "Root Certificates" are stored. Correct ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment