Decode packets encrypted with TLS/SSL

[fabianski7]

The HttpCanary app uses its own certificate to capture encrypted data from connections. Maybe this is the best way to do it?

it also uses the opensource library NetBare

[emanuele-f]

As far as I know, providing a custom certificate to do SSL mitm is the only way to implement this in android with no root. I recall that some new android version does not allow this. Moreover, it only works for apps which do not employ SSL certificate pinning.

[nitanmarcel]

@emanuele-f HttpCanary got a way around the SSL certificate pinning using Parallel Space app. Maybe you are able to find the same?

I'm using your app mainly because the Parallel Space version used by HttpCanary doesn't support work with Android 10 and above unfortunately

[emanuele-f]

Thank you for the pointers! More information on this methodology here AdguardTeam/AdguardForAndroid#1675

[nitanmarcel]

No problem. I hope we'll see something like this in the future with the app. Lately I've found myself using an emulator and HttpCanary but I would like to use my own phone for all of this.

[emanuele-f]

Temporary ignoring the android certificate limitations, I'm investigating how to integrate the mitm proxy functionality. I cannot find any usable library to do this, all the projects are meant for standalone use. Reimplementing it is pointless as it is a complex task already addressed by other people. Modifying and existing program is a possibility however not optimal as it forks the code. Another option would be to rely on mitmproxy for the decoding stuff and find a way to relay the packets to it.

[emanuele-f]

@nitanmarcel, @hardcoresecz, @fabianski7 the git version of PCAPdroid now supports sending the TLS packets to a remote mitmproxy for decryption. A modified mitmproxy is required right now to make it work, check out https://github.com/emanuele-f/mitmproxy. Will provide detailed documentation soon. In the meantime, in order to enable the TLS decryption:

1. compile latest PCAPdroid from github (will soon provide a new release for the download)
2. on a remote server, run mitmproxy --mode tunnel
3. import the CA certificate ~/.mitmproxy/mitmproxy-ca-cert.cer on the android phone. It is usually needed to rename it to .crt.
4. start PCAPdroid and configure the TLS decryption in the settings. All the TCP traffic on port 443 will be sent to the configured mitmproxy.

The decryption will only work if the app for which you are doing mitm (let's call this target app) trusts your user certificate, which only happens if at least one of these preconditions is satisfied:

• Android version of the device is less than 7 (Nougat)
• The target SDK version of the target app is less than 24
• The target app android manifest states that it trusts user certificates (see https://developer.android.com/training/articles/security-config#base-config)
• You can install system certificates. This is only possible on rooted devices / android emulators.

If you cannot satisfy any of the preconditions, there are still two possibilities:

Option 1:
Use the modified Parallel Space (only for Android < 10). Parallel Space is basically a virtualization app. We can say that it impersonates the target app, so any action taken by the app is instead performed by Parallel Space. HttpCanary bundles a version of Parallel Space with a target SDK version equal to 23. This means that the modified Parallel Space can run the target app with SDK version equal to 23, thus making it trust user certificates. Any virtualization app with appropriate SDK version should do the trick, so I will investigate if any available open source virtualization apps work so that the android 10 limitation can be removed. Right now, if you have android < 10, you can use the Parallel Space from HttpCanary together with PCAPdroid and it works.

Option 2:
Get the apk file for the target app, unpack it, modify the manifest to trust user certificates, repack it. The following program should be able to automate the process: https://github.com/shroudedcode/apk-mitm .

[emanuele-f]

The new release with the TLS decryption is out! Check out https://github.com/emanuele-f/PCAPdroid/releases/tag/v1.2.6 . It will be also available in the play store soon. The detailed information about TLS decryption is available at https://emanuele-f.github.io/PCAPdroid .

@nitanmarcel I've found that VirtualXposed works as a replacement of ParallelSpace and it should support android 10 too! Can you confirm this? Please check out the documentation at https://emanuele-f.github.io/PCAPdroid .

[nitanmarcel]

The new release with the TLS decryption is out! Check out https://github.com/emanuele-f/PCAPdroid/releases/tag/v1.2.6 . It will be also available in the play store soon. The detailed information about TLS decryption is available at https://emanuele-f.github.io/PCAPdroid .

@nitanmarcel I've found that VirtualXposed works as a replacement of ParallelSpace and it should support android 10 too! Can you confirm this? Please check out the documentation at https://emanuele-f.github.io/PCAPdroid .

Awesome, I'll try it and reply back as fast as I can

[nitanmarcel]

@emanuele-f Hi! I've tried using VirtualXposed and I couldn't bypass certificate pinning using it. Also VirtualXposed doesn't look at the right app for this since I've add some problems with it in refusing to launch some apps.

@emanuele-f Sorry for what I said above. I've had to do some extra setting to be able to connect to the mitmproxy.

I got one problem with VirtualXposed. Some apps can't run at all they even fail to launch or they crash soon after launching. And about the packet decryption I could succesfully bypass the certificate pinning using VirtualXposed.

[emanuele-f]

I got one problem with VirtualXposed. Some apps can't run at all they even fail to launch or they crash soon after launching. And about the packet decryption I could succesfully bypass the certificate pinning using VirtualXposed.

Thank you for your tests. There are some open crashes reported for VirtualXposed: https://github.com/android-hacker/VirtualXposed/issues?q=is%3Aissue+is%3Aopen+crash . I guess app virtualization is not an easy topic!

Do you have other suggestions about the TLS decryption in PCAPdroid? Does it fit your needs?

[nitanmarcel]

I got one problem with VirtualXposed. Some apps can't run at all they even fail to launch or they crash soon after launching. And about the packet decryption I could succesfully bypass the certificate pinning using VirtualXposed.

Thank you for your tests. There are some open crashes reported for VirtualXposed: https://github.com/android-hacker/VirtualXposed/issues?q=is%3Aissue+is%3Aopen+crash . I guess app virtualization is not an easy topic!

Do you have other suggestions about the TLS decryption in PCAPdroid? Does it fit your needs?

Well I can manage to work with the broken VirtualXposed if I need but I'm switching to an Android 11 custom ROM which will have root so would be nice to also add a tutorial on how to add the certificate as a system certificate in Android 7 and above for rooted devices.

[emanuele-f]

You can follow these instructions, I'll add a link into the docs. https://docs.mitmproxy.org/stable/howto-install-system-trusted-ca-android

[nitanmarcel]

You can follow these instructions, I'll add a link into the docs. https://docs.mitmproxy.org/stable/howto-install-system-trusted-ca-android

I just thought of that and I'm concerned it will trigger safety-net detections that will break some banking apps. Thankfully I've found a way to do this using Magisk which bypasses the check.

I've found this module, I haven't tested yet but from what I've read from the repo it should work:

https://github.com/NVISO-BE/MagiskTrustUserCerts

[nitanmarcel]

Thinking about it, is possible for mitmproxy to autogenerate the system certificate and pack it in a Magisk zip file?

[nitanmarcel]

https://topjohnwu.github.io/Magisk/guides.html

[emanuele-f]

I just thought of that and I'm concerned it will trigger safety-net detections that will break some banking apps. Thankfully I've found a way to do this using Magisk which bypasses the check.

Good, if you can find some detailed link I will add it to the docs. You can also provide a pull request if you prefer (see https://github.com/emanuele-f/PCAPdroid/blob/gh-pages/tls_decryption.md).

Thinking about it, is possible for mitmproxy to autogenerate the system certificate and pack it in a Magisk zip file?

In the docs I mention root approaches for completeness but they are not the main focus of PCAPdroid. Certificate generation may be integrated but no plan for magisk specific support.

[koh-osug]

I have updated the LittleProxy-mitm which is now using the maintained version of LittleProxy

I was testing it on Android 11 and Android 8 using TLS 1.3 and TLS 1.2. It is using Consrypt which is the internal crypto provider backed by boringssl in Android.

I have created an example application running a proxy on localhost port 9092.

The app is just a few MBs unshrinked. When using proguard and shrinking the app size should be even less. I would be interested to use this as a built-in MITM proxy instead of the remote one. What are the steps to integrate it as a tunnel?

Interesting for me would be also a mode allowing to connect to the proxy using socket / HTTP directly, so that the proxy alone could customized, e.g. in my case I have to be able to add client certificates from different sources.

[emanuele-f]

I have updated the LittleProxy-mitm which is now using the maintained version of LittleProxy

I can see only 1 commit in the new repo and no community issues, this cannot be considered a maintained app!

I was testing it on Android 11 and Android 8 using TLS 1.3 and TLS 1.2. It is using Consrypt which is the internal crypto provider backed by boringssl in Android.

Other than browsers, have you tested it with other apps? What method are you using to bypass the new certificate security which prevents apps from trusting user certificates?

I have created an example application running a proxy on localhost port 9092.

The app is just a few MBs unshrinked. When using proguard and shrinking the app size should be even less.

Please consider that PCAPdroid is currently very small (less than 5 MB) thanks to its minimal dependencies.

I would be interested to use this as a built-in MITM proxy instead of the remote one. What are the steps to integrate it as a tunnel?

The tunnel terminology I used may be a little misleading. You need the following:

• A way to instruct the proxy code about the original destination IP and port of the connection. sock5 provides a standard way to do this.
• If you want to show the unencrypted data in PCAPdroid, this will take some efforts as you need to modify the vpnproxy.c core code to read the decrypted data from the proxy instead of the raw packets. I would consider this step 2. But first, you should convince me of the integration!

Interesting for me would be also a mode allowing to connect to the proxy using socket / HTTP directly, so that the proxy alone could customized, e.g. in my case I have to be able to add client certificates from different sources.

This will be implemented with socks5 proxy.

[koh-osug]

I have updated the LittleProxy-mitm which is now using the maintained version of LittleProxy

I can see only 1 commit in the new repo and no community issues, this cannot be considered a maintained app!

I was just referring to the LittleProxy fork, which is maintained, the LittleProxy-mitm is the first updated version focusing an Android and had no chance to mature, yet. It has only a few classes which can also be integrated into PCAPdroid directly which would make more sense I guess, because like stated below the proxy on Android would only work with patched applications and makes only sense together with a VPNService on Android. LittleProxy-mitm on a PC as a standalone project would make sense but because of the user CA restrictions and heterogeneous proxy usage of Android applications it is of limited use.

I was testing it on Android 11 and Android 8 using TLS 1.3 and TLS 1.2. It is using Consrypt which is the internal crypto provider backed by boringssl in Android.

Other than browsers, have you tested it with other apps? What method are you using to bypass the new certificate security which prevents apps from trusting user certificates?

I have tested it with Chrome, I have patched Opera to see if it works with a modified network security configuration including the MITM CA (using apktool). Firefox is using its own network stack and does ignore the network security configuration. Some other apps are not honoring the proxy settings of Android. Because of this my goal is to use a VPNService which will be used by all applications transparently. The proxy alone cannot work without patching the applications and needs the VPNService.

I have created an example application running a proxy on localhost port 9092.

The app is just a few MBs unshrinked. When using proguard and shrinking the app size should be even less.

Please consider that PCAPdroid is currently very small (less than 5 MB) thanks to its minimal dependencies.

A version with the included proxy should also stay clearly under 10 MB. If anyone has dramatic size restrictions two versions could be offered with and without the proxy.

I would be interested to use this as a built-in MITM proxy instead of the remote one. What are the steps to integrate it as a tunnel?
The tunnel terminology I used may be a little misleading. You need the following:

A way to instruct the proxy code about the original destination IP and port of the connection. sock5 provides a standard way to do this.
If you want to show the unencrypted data in PCAPdroid, this will take some efforts as you need to modify the vpnproxy.c core code to read the decrypted data from the proxy instead of the raw packets. I would consider this step 2. But first, you should convince me of the integration!

After several days of research this was the only option I found, so it is a question if a convenient built-in integration is the goal or the more complex setup using a different computer and using a modified version of mitmproxy should be the only option.

[emanuele-f]

Moving the discussion to #57. I've clarified the requirements and current candidates. To answer your question, built-in TLS decryption is a nice feature to have in PCAPdroid, however not a priority. If you are willing to integrate it, your contribution is welcome.

[geekyayush]

Is there any guide for decoding the encrypted packets using PCAPdroid?
I'd appreciate if anyone can share.

[emanuele-f]

https://emanuele-f.github.io/PCAPdroid/tls_decryption