Add TLS config mirroring

[fedosgad]

Description

Modify tlsconfig.py to mirror client's ciphersuites to server

Checklist

• I have updated tests where applicable.
• I have added an entry to the CHANGELOG.

[mhils]

Thanks for the initial PR! 🍰 Regarding your questions from Slack

Chrome sends GREASE ciphersuites. Can pyOpenSSL be somehow tricked into sending them without patching & recompiling?

I think the question is if OpenSSL can, pyOpenSSL we can make work somehow. Unfortunately I don't think OpenSSL will cooperate. On a more positive note, at least JA3 hashes seem to ignore GREASE entirely.

OpenSSL sends TLS_FALLBACK_SCSV ciphersuite to signal some protection against downgrading. Can I turn it off? (https://wiki.openssl.org/index.php/SSL_MODE_SEND_FALLBACK_SCSV states that unsafe modes must be turned off, but that's against the whole mirroring idea)

That question is slightly confusing to me - when do you see SSL_MODE_SEND_FALLBACK_SCSV? I thought this would only show up on retries if the application sets it explicitly.

[mhils]

Please don't alias modules unless it's absolutely necessary.

[mhils]

I have a slight preference for naming the default "secure", because one benefit of not mirroring is that we can avoid insecure/old ciphers. Naming it "secure" makes the tradeoff more clear.

[fedosgad]

Valid (I'll rename it), but consider reviewing default cipher list - I don't think DES-CBC3-SHA (at least) counts secure anymore

[mhils]

The current list is what Mozilla recommends when old clients might be present (https://ssl-config.mozilla.org/#config=old). You have a valid point though. How about we introduce secure/old/mirror, with secure being turned on by default? We can use the intermediate and old settings on https://ssl-config.mozilla.org/ for secure and old respectively.

[mhils]

This won't work with concurrent connections, tls_clienthello may be called for two different connections before the corresponding tls_start_server calls are made. The proxyauth addon has an example of how you can keep state for a connection around:

mitmproxy/mitmproxy/addons/proxyauth.py

Line 26 in 65773cc

self.authenticated: MutableMapping[connection.Client, Tuple[str, str]] = weakref.WeakKeyDictionary()

[mhils]

I like the idea of limiting options. However, I'd prefer if we could use typing.Literal for this:

TlsStrategy = Literal["secure","mirror"]
# getting all options:
choices=typing.get_args(TlsStrategy)

This is a bit simpler to handle than enums, in particular you don't run into issues like omitting .name (see below).

[mhils]

First, I think this line is missing a .name. Second, please use ==, we shouldn't rely on string interning here. While it's correct to compare Enums with is the same does not apply for their name (which is a plain string). Small strings happen to be inlined in Python, but that's nothing that should be relied upon.

[mhils]

An easier way to accomplish this is to say ciphersuite_names.get(num, None). :-)

[mhils]

Let's just call SSL._lib.SSL_CTX_set_ciphersuites(ssl_ctx._context, ciphersuites) directly here.

[fedosgad]

Why? (Sorry, but I'm not a python coder, so I'd like to know)
What if pyopenssl merges the pull request where this one is from?

[mhils]

I dislike

ssl_ctx.set_ciphersuites = cs.set_ciphersuites

because it monkeypatches a new method onto an existing object. That can be very confusing to debug down the line, especially when pyOpenSSL merges a slightly different implementation for set_ciphersuites. You would expect ssl_ctx.set_ciphersuites to be what pyOpenSSL's provides, but we'd be secretly overwriting it here then.

It's more straightforward if we skip the pyOpenSSL wrapper and directly interact with the OpenSSL API:

SSL._openssl_assert(
    SSL._lib.SSL_CTX_set_ciphersuites(ssl_ctx._context, ciphersuites) == 1
)

[LicketySpliket]

That question is slightly confusing to me - when do you see SSL_MODE_SEND_FALLBACK_SCSV? I thought this would only show up on retries if the application sets it explicitly.

Not OP, but on my end SCSV suites are sent by pyOpenSSL on the client hello. Specifically,
TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff) is sent even when not included in the cipher list.

[fedosgad]

That's exactly like @LicketySpliket says - it always appears at the end of the ciphersuites' list, even if I comment it out in map (i.e. it's not copied from client request). Tested it by visiting https://howsmyssl.com with Chrome and Firefox with and without mitmproxy. Also, all requests using curl always send this CS (even without mitmproxy). I suppose this is from OpenSSL itself.

[mhils]

Thanks. I'm as smart as anyone here, but https://www.openssl.org/docs/manmaster/man3/SSL_get_secure_renegotiation_support.html#SECURE-RENEGOTIATION has two options that could at least be tried out.
SSL._lib.SSL_OP_LEGACY_SERVER_CONNECT and SSL._lib.SSL_OP_ALLOW_UNSAFE_LEGACY_RENEGOTIATION are both available via cryptography.

[EndUser509]

I am also interested in this feature. I hope this will defeat TLS fingerprinting like cloudflare does which is annoyance for me. One question though: Is it enough to just clone the cipher list or are the extensions also part of the fingerprint?
I checked the current openssl source and in ssl_cipher_list_to_bytes() the addition of SSL3_CK_SCSV (openssl name for TLS_EMPTY_RENEGOTIATION_INFO_SCSV) is hard-coded if renegotiation is off. And renegotiation has to be off for HTTP/2 per RFC7540. Can we just turn on renegotiation and get away with it?
I think the best option is to post-process the client hello generated by the code above and remove that signaling cipher from the list and to additional corrections if needed. In the same process one could also verify if the mirroring could be successfully done and issue a warning if not. Any opinions?

[mhils]

One question though: Is it enough to just clone the cipher list or are the extensions also part of the fingerprint?

That depends on the fingerprint being used. https://github.com/salesforce/ja3 is common, although I'd suspect that Cloudflare uses their own thingy.

Can we just turn on renegotiation and get away with it?

(I know as much as you do here)

I think the best option is to post-process the client hello generated by the code above and remove that signaling cipher from the list and to additional corrections if needed. In the same process one could also verify if the mirroring could be successfully done and issue a warning if not. Any opinions?

I'm afraid we kind of need to get our TLS library (OpenSSL curently) to emit the right thing, otherwise the crypto operations will fail later (you can't just modify the ClientHello and expect to get away with it). I'm afraid the options are 1) coerce OpenSSL to do what we want (not sure if possible), or 2) switch to rustls as an alternative TLS backend and modify it to our needs. I refuse to ship a custom patched version of OpenSSL (too much pain), but PyO3 is actually very awesome to use.

[EndUser509]

One question though: Is it enough to just clone the cipher list or are the extensions also part of the fingerprint?

That depends on the fingerprint being used. https://github.com/salesforce/ja3 is common, although I'd suspect that Cloudflare uses their own thingy.

Thanks for the link. That implementation checks more than the cipher list so that is a no then and this PR alone will not solve anything.

Can we just turn on renegotiation and get away with it?

(I know as much as you do here)

With the answer from above this question became irrelevant.

I think the best option is to post-process the client hello generated by the code above and remove that signaling cipher from the list and to additional corrections if needed. In the same process one could also verify if the mirroring could be successfully done and issue a warning if not. Any opinions?

I'm afraid we kind of need to get our TLS library (OpenSSL curently) to emit the right thing, otherwise the crypto operations will fail later (you can't just modify the ClientHello and expect to get away with it).

Are you sure? As long as the modification of the client_helo does not advertise a feature that openssl does not support, this should work? The frame is AFAIK not cryptographically secured so any functional modification such as these should pass:

• Remove or reorder ciphers, extensions, curves, etc.
• Add GREASE
• Add ciphers to the list that are unlikely to be picked because better ones are at the head of the list.

I'm afraid the options are 1) coerce OpenSSL to do what we want (not sure if possible), or 2) switch to rustls as an alternative TLS backend and modify it to our needs. I refuse to ship a custom patched version of OpenSSL (too much pain), but PyO3 is actually very awesome to use.

2. sounds like a larger project. If 1) is somehow feasible that would be the best.

[mhils]

Are you sure? As long as the modification of the client_helo does not advertise a feature that openssl does not support, this should work? The frame is AFAIK not cryptographically secured so any functional modification such as these should pass:

The entire ClientHello directly feeds into the transcript hash (https://datatracker.ietf.org/doc/html/rfc8446#section-4.4.1), so modifications will be detected.

[EndUser509]

Are you sure? As long as the modification of the client_helo does not advertise a feature that openssl does not support, this should work? The frame is AFAIK not cryptographically secured so any functional modification such as these should pass:

The

Are you sure? As long as the modification of the client_helo does not advertise a feature that openssl does not support, this should work? The frame is AFAIK not cryptographically secured so any functional modification such as these should pass:

The entire ClientHello directly feeds into the transcript hash (https://datatracker.ietf.org/doc/html/rfc8446#section-4.4.1), so modifications will be detected.

You are right, my memory was a little rusty from the times where cipher downgrade attacks were still possible. :)

However, I did look at this a little more. Maybe it is possible to alter the client hello message by calling ssl3_init_finished_mac() and then ssl3_finish_mac(altered client hello) to restart the MAC. This is internally also done in response to a HelloRetryRequest. I have not checked if the two functions are accessible from python though.

If that does not work, I think it will get tough, openssl just does not give enough control over the client hello construction process. And the use of TLS_EMPTY_RENEGOTIATION_INFO_SCSV is the way openssl solves the renegotiation vulnerability. iOS uses a regegotiation_info extension and you cannot get openssl to send that: openssl/openssl#8143

[fedosgad]

Talking about fingerprinting, there's also Cloudflare's MALCOLM (https://github.com/cloudflare/mitmengine & https://malcolm.cloudflare.com/) which, as I get it, uses technique other than JA3
Anyway, I can say from my experience that ciphersuite list matters most in fingerprinting, but not everything (I've seen websites that also check extensions present).

[mhils]

Thanks for the pointer @fedosgad! I think it's not completely unintentional that OpenSSL does not support spoofing browser ClientHellos. Making rustls work would be a very fun undertaking, but also a considerable effort.

[GunGunGun]

Talking about fingerprinting, there's also Cloudflare's MALCOLM (https://github.com/cloudflare/mitmengine & https://malcolm.cloudflare.com/) which, as I get it, uses technique other than JA3 Anyway, I can say from my experience that ciphersuite list matters most in fingerprinting, but not everything (I've seen websites that also check extensions present).

From my testings, the one actually matters the most is Extensions, I tried to match browser's ciphersuites, signature algorithms, ec point formats, supported groups and websites that use Cloudflare always return 403 status error because OpenSSL can't really spoof Extensions.

You can easily test with curl-impersonate, try to modify User-Agent, even ciphersuite of Chrome and it still passes Cloudflare's checks, because Extensions are still matched with Chrome. In fact I changed heavily chrome_104.config to:

--ciphers TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-CHACHA20-POLY1305

--http2
--false-start
--compressed
--tlsv1.2
--no-npn
--alps
--cert-compression brotli
--location

And chrome_104.headers to:

sec-ch-ua: "Chromium";v="104", " Not A;Brand";v="99", "Google Chrome";v="104"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:102.0) Gecko/20100101 Goanna/5.2 Firefox/102.0 PaleMoon/31.3.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

Test cmd (this website uses Cloudflare's anti bot settings):

curl_chrome104.bat https://alternativeto.net

And it stills works, even with heavily utterly modified Ciphersuite and User-agent.

Now, change chrome_104.config again to:

--ciphers TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,ECDHE-ECDSA-AES128-GCM-SHA256,ECDHE-RSA-AES128-GCM-SHA256,ECDHE-ECDSA-AES256-GCM-SHA384,ECDHE-RSA-AES256-GCM-SHA384,ECDHE-ECDSA-CHACHA20-POLY1305

--http2
--false-start
--compressed
--tlsv1.2
--alps
--cert-compression brotli
--location

Run above cmd again, and Cloudflare will return 403 because I removed --no-npn to add NPN extension to Chrome, and that's a death sentence because I changed Extensions.

So my conclusions, for Cloudflare:

• Ciphersuite doesn't really matter
• User-agent doesn't matter
• Extension matters

Link: https://github.com/depler/curl-impersonate-win

[fedosgad]

Closing this PR because:

1. I don't see a way to make OpenSSL fully mimic some given fingerprint.
2. I don't want this old PR to consume developers' time.
3. I made a tool specifically for mimic tasks, albeit not as comfortable as mitmproxy - https://github.com/fedosgad/mirror_proxy/.

Thanks everyone who participated in discussion and special thanks to @mhils for code review and valuable hints about it!