Built-in TLS Decryption

[emanuele-f]

At the current status, PCAPdroid requires an external proxy in order to decrypt TLS packets. On the other hand, a built-in solution would provide the following advantages:

• No need to have a PC to decrypt TLS
• PCAPdroid could show the decrypted payload directly into the app. This is related to Add basic payload information #38.
• PCAPdroid could produce a decrypted decryptable PCAP

Requirements

• Should support TLS 1.3
• Should not impact the main PCAPdroid development. TLS decryption is a secondary topic in PCAPdroid, so this feature should be supported by an active third party
• Should comply to the F-Droid policy (e.g. do not bundle any binary)
• The solution should be light, primary in terms of device resources (cpu, memory) and possibly in terms of dependencies and APK size

Candidates

• LittleProxy-mitm/LittleProxy is an HTTP proxy. It fits the purpose of a ready-to-use library, however depends on netty, which is big dependency.
• nitmproxy is a self contained SOCKS/HTTP proxy also using netty. SOCKS offers the possibility to decrypt non-HTTPS traffic.

Other approaches:

• NetBare: unmaintained, requires heavy modifications to only extract the proxy code, does not support TLS 1.3, it has the advantage of using java.nio for networking, which fits the minimal dependencies requirement
• any native C/C++ library could be integrated

User Experience

Users will have an option to enable the built-in TLS decryption by a toggle. The old mitmproxy option will be replaced with a more generic ability to use socks5 proxy, which is independent from the TLS decryption. Once enabled, the TLS decryption will happen under the hoods; android proxy settings will be untouched. It will have the following effects:

• In the connections view, the TLS connections data will be replaced by decrypted protocol data, e.g. HTTPS will be decrypted and the HTTP request/reply will be shown
• The PCAP will also contain the decrypted payload data in place of the original TLS data The traffic could be dumped in the PCAPNG file format, which contains a "Decryption Secrets Block" to store the decryption keys.

In essence TLS will be decapsulated and its inner data shown.

Roadmap

The built-in decryption is now implemented via the PCAPdroid-mitm addon.

• Implement socks5 client in PCAPdroid (Can you add a function to transfer all TCP data to the remote mitmproxy (or other proxy)? #44)
• Build the mitm proxy addon
• Verify that embedded mitm proxy properly proxies the packets received by the socks5 interface
• Show the decrypted connection in PCAPdroid
• Report rejected certificates warnings from mitmproxy in the app
• There are some TLS connections which are not decrypted, possibly fix/report to the user why this happens
• Provide certificate installation via Intent on Android < 11
• Harden the proxy via the SOCKS5 authentication
• Fix export of SSLKEYLOG
• Improve matching of HTTP request/replies in the UI (e.g. highlight row when clicking "jump" icon, or sort connections)
• Documentation, updates docs on methods to bypass untrusted user certs (on Android 7, e.g. magisk trust user certs, bromite chrome://flags), pinning (LSposed module sslunpinning) and Certificate transparency (avoid using a system cert or use magisk hide) - see the docs
• Suggest users to disable Autostart or similar software which can prevent the mitm addon execution

[emanuele-f]

With the purpose of minimizing the dependencies, here is a quick overview of the LittleProxy-mitm depedencies (max depth 3):

- koh-osug/LittleProxy-mitm
  - bcpkix-jdk15to18
  - bcprov-jdk15to18
  - conscrypt-openjdk-uber
  - commons-io
  - slf4j-log4j12
  - mrog/LittleProxy
    - guava
    - commons-lang3
    - netty-all
    - barchart-udt-bundle (UDT transport protocol)
    - slf4j-api

@koh-osug can you please clarify the need for the additional dependencies of LittleProxy-mitm over LittleProxy? E.g. jdk15to18 and conscrypt. Regarding the project structure, I think it's better to keep LittleProxy-mitm in a separate code base, unless it requires a tight integration with PCAPdroid.

[koh-osug]

• I haved removed the commons-io dependency.
• conscrypt-openjdk-uber must be excluded when using the library under Android, it is only required in Java-SE, Android is already using this internally as crypto provider.
• bcpkix-jdk15to18 and bcprov-jdk15to18 are needed for creating the MITM certificates.
• Under Android the barchart-udt-bundle dependency must be excluded, it is optional and not supported and not needed. I have patched the latest version of littleproxy to make it optional under Android with a merge request and it was already merged.
• I guess, I cannot influence the LittleProxy maintainer to remove guava or commons-lang3, these 2 libraries are convenient, but usually only a subset of the functions are needed.

I will check with my Android test project how much is left after using the Android shrinker. Usually almost all from the utilitiy libraries and major parts of bouncy castle should get removed resulting in just a few HD floppy disks.

[emanuele-f]

Great! I think LittleProxy should be run as a thread in PCAPdroid and communicate with it via the socks5 interface (will I will take care of the socks5 migration asap). This will allow PCAPdroid to carry on the TLS decryption asynchronously while processing the network packets in parallel.

I've added a roadmap for this feature in the issue description above, we can work on a branch until it is complete. You can proceed with the integration. Once the embedded LittleProxy works correctly, I will take care of the final step to show the decrypted data in PCAPdroid/in the PCAP. For this I will need a call API in LittleProxy-mitm to read the decrypted data back in PCAPdroid. The API should expose: the decrypted data byte[], the direction of the data (from/to the internet) and the connection identifier (the 5 tuple, or at least proto, dst_ip, src_port, dst_port)

[koh-osug]

I have checked the sizes after shrinking:

• BouncyCastle is just using 261 kb
• guava was reduced to 146 kb
• netty is 812 kb
• LittleProxy-mitm itself is 71 kb

[koh-osug]

Filtering the HTTP request seems to be possible with LittleProxy. I hope the filter is the one seeing the plain text. How will you map the plain text to the original requests? I can record the plain text, but somehow then later you have to know which plain text corresponds to which encrypted text.

[emanuele-f]

I have checked the sizes after shrinking

1.3 MB is a good result, I think we can settle with this. Great work!

Filtering the HTTP request seems to be possible with LittleProxy. I hope the filter is the one seeing the plain text. How will you map the plain text to the original requests? I can record the plain text, but somehow then later you have to know which plain text corresponds to which encrypted text.

The idea is the following: when LittleProxy is started I will register a callback (via a java interface). Whenever the proxy decrypts a packet, it will call the callback with the decrypted data and the connection information (original 5 tuple). With the 5 tuple I can retrieve the original connection and set the plain text data.

In b029895 I've implemented the SOCKS5 proxy client. This replaces the old "tunnel" mode. You should now be able to redirect the TCP packets to LittleProxy by using SOCKS5.

Edit: another step is required in order to make this work with an embedded LittleProxy. Since LittleProxy will make connections on his own, we must exclude these connections from the VPN, otherwise they will go back into the VPNService in a loop. To do this, the VpnService protect api must be called on every socket open by LittleProxy. In order to keep LittleProxy clean, I'd suggest to do this by registering a callback at LittleProxy startup via a java interface. Of course I'm assuming that LittleProxy provides an api to extract the raw socket from netty.

[koh-osug]

I will have a look into this. My first concern is if LittleProxy is actually supporting SOCKS5. I'm not familiar with the protocol, but what I see is that it needs a HTTP CONNECT call to trigger the proxy.

[koh-osug]

I'm investigating this one as alternative: https://github.com/chhsiao90/nitmproxy
It supports socks and can also be used for SMTP / IMAP with TLS then.

[koh-osug]

I had an issue with nitmproxy because the addresses received from SOCKS have been IP addresses, no hostnames anymore. I added a handler to extract the Server Name Indication and this looks good now. I created a fork and check its stability. LittleProxy might be maintained, but it is focusing too much on HTTP, nitmproxy supports more protocols and has the MITM feature is included. I have tested in under Linux and by connecting from Android to it.

[emanuele-f]

Have you tested it with TLS 1.3 and HTTP2? A SOCKS proxy opens up more possibilities so I also encourage this choice.

[koh-osug]

I have tested the page "https://tools.keycdn.com/http2-test" with FF usign the HTTP2 protocol. The web page is working.
I used the web page https://resources.duda.co/tls-1-3 to check for TLS 1.3. This was also working.
Maybe some unit test could be added, too.

[emanuele-f]

Great! Please provide a pull request/branch to review when ready! If you want a more interactive technical discussion you can reach me on the telegram group