Detect HTTPS interception #1430

mholt · 2017-02-13T17:20:11Z

This is a WIP branch which closes #1420. It makes it possible for Caddy to detect when it is likely that the TLS connection is being intercepted. This is possible because TLS proxies are mostly careless. It works by comparing the properties of the TLS Client Hello message to what is expected of the client indicated by the User-Agent HTTP header. If there is a mismatch, a MITM is likely.

The heuristics used here are a partial implementation of The Security Impact of HTTPS Interception by Durumeric, Halderman, and others.

I would like people to try this out -- you will need to compile this branch with Go 1.8.

This PR also adds:

{{.IsMITM}} template action that returns true or false.
{mitm} placeholder which can be used in the Caddyfile to initiate redirects or add the value to a log entry. It returns likely, unlikely, or unknown.

TODO:

Write tests
Flush out the Safari heuristics
Seek clarification on the IE/Edge heuristics as they seem to overlap with Chrome
Reading the TLS Client Hello in Accept() is currently blocking, I think. Verify if this is the case and, if it is, find a way to read the Hello without blocking other connections...
A way to close the connection right away if a MITM is suspected (will be exposed via Caddyfile directive later if this feature is useful)

An easy way of testing this is to use Firefox to load a HTTPS page, then use the Dev Tools to "Edit and Resend" the HTTP request, changing the User-Agent header to replace "Firefox" with something like "Chrome" or any other browser.

I would like to encourage anyone to test this out and submit feedback, this is a highly experimental feature but one I feel could be important. Any other collaborators that would like to participate in reviewing, please do!

@FiloSottile

Special thanks to @FiloSottile for guidance with the custom listener.

elcore

It's looking good at the first glance. Well done Matt!!!

P.S.: I will take a deeper look in the next few days/weeks

As far as I can tell, reading the ClientHello during Accept() prevents new connections from being accepted during the read. Since Read() should be called in its own goroutine, this keeps Accept() non-blocking.

elcore · 2017-02-14T18:16:06Z

caddyhttp/httpserver/mitm.go

+	// cipher suites missing from the crypto/tls package,
+	// in no particular order here
+	TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 = 0xcca9
+	TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256   = 0xcca8


ChaCha20 cipher suites are not missing from the crypto/tls package.

https://tip.golang.org/pkg/crypto/tls/#pkg-constants

TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 uint16 = 0xcca8 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 uint16 = 0xcca9

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 is not missing in the crypto/tls package.

Oh, thanks.

No problem 😄, in my spare time I will try to review this PR

tobya · 2017-02-15T15:52:08Z

Not sure if this shows something or nothing, but i tested by editing firefox request as suggested and got expected results.

Then I connected using my phone so using the mobile network without editing anything. Got the following results

178.167.254.199 - [15/Feb/2017:15:48:19 +0000] 'GET  /list.php /list.php  - HTTP/2.0 200 51060' https unlikely
178.167.254.199 - [15/Feb/2017:15:48:23 +0000] 'GET  /favicon.ico /favicon.ico  - HTTP/2.0 404 58442' https likely

Would there be a reason that the file I requested list.php is unlikely but the favicon.ico (which doesnt exist) would be likely?

All other requests on this website over mobile network are unlikely. I even tried requesting a non-existant file (404) and got unlikely. Requesting favicon.ico a second time give a unlikely

mholt · 2017-02-15T18:09:50Z

The heuristics used in the research will be published probably no sooner than March 1, 2017, but I may merge this PR before then and then iterate on this feature later at that time.

mholt · 2017-02-15T18:12:27Z

@tobya Interesting... can you add {remote}:{port} to your log format and see if they are the same? Also, since you're building this from source, if you dump the request headers (fmt.Printf("Headers: %+v", r.Header) might be helpful too. Also, which browser are you using on mobile?

tobya · 2017-02-15T22:16:05Z

@mholt {remote} and {port} are there (just apart)

178.167.254.199:51060 /list.php https unlikely
178.167.254.199:58442 /favicon.ico' https likely

The ports are differnt.

I can reproduce consistently with 3G on my mobile on Chrome for iOS. This is output of headers.

178.167.254.34:58772 - [15/Feb/2017:21:59:47 +0000] 'GET asite4.bcs.cookingisfun.ie  /list.php /list.php  - HTTP/2.0 200 ' https unlikely

Headers: map[Accept-Encoding:[gzip, deflate] Cookie:[_ga=GA1.2.1285204035.1421831904; __utma=71723596.1285204035.1421831904.1478764756.1478864814.78;     __utmz=71723596.1478705289.76.15.utmcsr=jobsforcooks.com|utmccn=(referral)|utmcmd=referral|utmcct=/employers/] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8] User-Agent:[Mozilla/5.0 (    iPhone; CPU iPhone OS 10_0_2 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.79 Mobile/14A456 Safari/602.1] Accept-Language:[en-ie]]

178.167.254.34:50453 - [15/Feb/2017:21:59:48 +0000] 'GET asite4.bcs.cookingisfun.ie  /favicon.ico /favicon.ico  - HTTP/2.0 404 ' https likely

Headers: map[User-Agent:[Mozilla/5.0 (iPhone; CPU iPhone OS 10_0_2 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.79 Mobile/14A456 Safari/602.1] Accept-Encoding:[gzip, deflate, sdch]     Accept-Language:[en-US,en;q=0.8]]

As you guessed the ports are different. This only happens on my mobile phone and only on Chrome, firefox for iOS and Safari for iOS dont cause this.

I have uploaded the log with headers for various calls with 3 differnt browsers https://gist.github.com/tobya/8ec9e4b6553264b6e2022b228a58f631

mholt · 2017-02-15T23:04:43Z

@tobya Thanks, this might be a flaw in how I'm guessing the browser from the User-Agent string. It probably thinks it is Safari and not Chrome because Chrome doesn't appear in the UA. 😕 That's annoying. What the heck is CriOS?

I don't have iOS so I can't test this. If you want to help fix it, put some print/log statements in the ServeHTTP of mitm.go and see which browser it thinks the client is based on the UA and let's see if it's getting it right, which I bet it's not. It doesn't explain why only some are false positives though... but maybe that will be clear later...

The different port means that your phone is opening two separate connections. But why, and why are they somehow different? No clue at this point.

If you add a check for "CriOS" (Chrome iOS? I have no idea) here in the UA checking logic of ServeHTTP of mitm.go, does that help with the false positive at all?

tobya · 2017-02-16T22:40:27Z

Should I move this to a seperate issue?

From my position of ignorance it seems it may be that Chrome on iOS is using a cipher that is in your list of ciphers not supported by Chrome specifically I output ext at this line and output

uint16=49188

which seems to be TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384

Would a fix to be to add CriOS as another specific browser and remove that cipher from the list?

mholt · 2017-02-16T23:42:37Z

@tobya This issue is fine, we're almost done I think.

I need from you:

The hex encoding of the TLS ClientHello (the current commit on this branch prints this out for you)
The exact browser and iOS versions

And yes, I think that is a reasonable course of action: add CriOS as an alternative to Chrome in our conditions and then remove that cipher from the list of exclusions. If you want to try that in your build too and see if it fixes it, that would be great!

And with those two pieces of information I asked you for I'll add this to our test corpus.

# Conflicts: # caddyhttp/httpserver/server.go

wendigo · 2017-02-17T11:44:31Z

LGTM :)

mholt · 2017-02-17T16:18:32Z

@wendigo Thanks. Have you used the "review" feature on Github before? Just so you know, you can leave a review with inline comments and express approval or request changes that way.

mholt added 2 commits February 13, 2017 10:09

WIP: Implement HTTPS interception detection by Durumeric, et. al.

5701dcb

Special thanks to @FiloSottile for guidance with the custom listener.

Add {{.IsMITM}} context action and {mitm} placeholder

8a6ab5f

mholt added help wanted 🆘 Extra attention is needed in progress 🏃‍♂️ Being actively worked on needs docs ✍️ Requires documentation changes needs tests 💯 Requires automated tests labels Feb 13, 2017

mholt requested review from tw4452852 and FiloSottile February 13, 2017 17:20

mholt self-assigned this Feb 13, 2017

elcore reviewed Feb 14, 2017

View reviewed changes

mholt added 3 commits February 13, 2017 23:49

Improve MITM detection heuristics for Firefox and Edge

cbf62a7

Add tests for MITM detection heuristics

fd323a3

Improve Safari heuristics for interception detection

7b7e1e4

mholt removed the needs tests 💯 Requires automated tests label Feb 14, 2017

Read ClientHello during first Read() instead of during Accept()

dc1af8f

As far as I can tell, reading the ClientHello during Accept() prevents new connections from being accepted during the read. Since Read() should be called in its own goroutine, this keeps Accept() non-blocking.

elcore reviewed Feb 14, 2017

View reviewed changes

mholt added 2 commits February 14, 2017 17:38

Clean up MITM detection handler; make possible to close connection

fbc1114

Use standard lib cipher suite values when possible

3328e94

mholt removed in progress 🏃‍♂️ Being actively worked on needs docs ✍️ Requires documentation changes labels Feb 15, 2017

Improve Edge heuristics and test cases

479fa6a

Refactor MITM checking logic; add some debug statements for now

30de2e4

This was referenced Feb 16, 2017

Use RequestURI when redirecting to canonical path. #1331

Merged

on a port using TLS, suppress the TLS alert on unexpected http requests #747

Closed

Store name of authenticated user in basicauth #1426

Merged

mholt removed the help wanted 🆘 Extra attention is needed label Feb 16, 2017

mholt added 4 commits February 16, 2017 17:48

Merge branch 'master' into mitmdetect

d76a906

# Conflicts: # caddyhttp/httpserver/server.go

Fix bug in MITM heuristic tests and actual heuristic code

8122d7c

Fix gofmt

e5ab9ca

Merge branch 'master' into mitmdetect

34e3a91

Remove debug statements; preparing for merge

3f49c57

wendigo approved these changes Feb 17, 2017

View reviewed changes

mholt merged commit 82cbd7a into master Feb 17, 2017

mholt deleted the mitmdetect branch February 17, 2017 21:09

This was referenced Feb 17, 2017

Fixed false positive of mitm on CRiOS Chrome on iOS #1448

Closed

Add burp mitm detection. #1454

Closed

This was referenced Jan 15, 2023

consider using ja3 to block bots komuw/ong#95

Closed

crypto/tls: add Extensions to ClientHelloInfo golang/go#32936

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect HTTPS interception #1430

Detect HTTPS interception #1430

mholt commented Feb 13, 2017 •

edited

Loading

elcore left a comment •

edited

Loading

elcore Feb 14, 2017

mholt Feb 14, 2017

elcore Feb 14, 2017

tobya commented Feb 15, 2017 •

edited

Loading

mholt commented Feb 15, 2017

mholt commented Feb 15, 2017 •

edited

Loading

tobya commented Feb 15, 2017

mholt commented Feb 15, 2017 •

edited

Loading

tobya commented Feb 16, 2017 •

edited

Loading

mholt commented Feb 16, 2017

wendigo commented Feb 17, 2017

mholt commented Feb 17, 2017

Detect HTTPS interception #1430

Detect HTTPS interception #1430

Conversation

mholt commented Feb 13, 2017 • edited Loading

elcore left a comment • edited Loading

Choose a reason for hiding this comment

elcore Feb 14, 2017

Choose a reason for hiding this comment

mholt Feb 14, 2017

Choose a reason for hiding this comment

elcore Feb 14, 2017

Choose a reason for hiding this comment

tobya commented Feb 15, 2017 • edited Loading

mholt commented Feb 15, 2017

mholt commented Feb 15, 2017 • edited Loading

tobya commented Feb 15, 2017

mholt commented Feb 15, 2017 • edited Loading

tobya commented Feb 16, 2017 • edited Loading

mholt commented Feb 16, 2017

wendigo commented Feb 17, 2017

mholt commented Feb 17, 2017

mholt commented Feb 13, 2017 •

edited

Loading

elcore left a comment •

edited

Loading

tobya commented Feb 15, 2017 •

edited

Loading

mholt commented Feb 15, 2017 •

edited

Loading

mholt commented Feb 15, 2017 •

edited

Loading

tobya commented Feb 16, 2017 •

edited

Loading