Mitm'ed HTTPS responses are not automatically deflated #9

chancancode · 2013-06-25T01:48:06Z

I'm trying to get the HTTPS MITM to work, and it looks like HTTPS responses are not automatically deflated before they are handed to the handlers.

elazarl · 2013-06-25T05:15:09Z

Can you provide a short snippet of code to reproduce?

On Tue, Jun 25, 2013 at 4:48 AM, Godfrey Chan notifications@github.comwrote:

I'm trying to get the HTTPS MITM to work, and it looks like HTTPS
responses are not automatically deflated before they are handed to the
handlers.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/9
.

chancancode · 2013-06-25T05:45:37Z

Here you go:

package main

import (
    "fmt"
    "github.com/elazarl/goproxy"
    "github.com/elazarl/goproxy/ext/html"
    "net/http"
    "strings"
)

func main() {
    proxy := goproxy.NewProxyHttpServer()

    proxy.OnRequest(goproxy.ReqHostIs("www.ietf.org:443")).HandleConnect(goproxy.AlwaysMitm)

    proxy.OnResponse(goproxy.DstHostIs("www.ietf.org")).Do(goproxy_html.HandleString(func(s string, ctx *goproxy.ProxyCtx) string {
        fmt.Printf("[HTTP] First line of body: '%s'\n", strings.SplitN(s, "\n", 2)[0])
        return s
    }))

    proxy.OnResponse(goproxy.DstHostIs("www.ietf.org:443")).Do(goproxy_html.HandleString(func(s string, ctx *goproxy.ProxyCtx) string {
        fmt.Printf("[HTTPS] First line of body: '%s'\n", strings.SplitN(s, "\n", 2)[0])
        return s
    }))

    panic(http.ListenAndServe(":8080", proxy))
}

Result:

% go run test.go
[HTTP] First line of body: '<!DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">'
2013/06/24 22:42:34 [002] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63600: connection reset by peer
2013/06/24 22:42:34 [003] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63601: connection reset by peer
2013/06/24 22:42:34 [004] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63602: connection reset by peer
2013/06/24 22:42:34 [005] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63603: connection reset by peer
2013/06/24 22:42:34 [007] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63605: connection reset by peer
2013/06/24 22:42:34 [006] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63604: connection reset by peer
[HTTPS] First line of body: '?ks۸?{~?7?ǌ%Jr???V???7N??Υ?ԁH?BL
                                                            Z?M|w??DҖb:7??|?????????Oǟ???}~B?q?ጜ~{vzD???~]??C??Rɞ??f??l?+d?^]??S?}|?>vT?ͮ?|g?????(???8???Å?3<???!?q?h챷,?1Q,JB?ء?^???å?b&ixN??g?!??٧L??g??iz&?k?:??!???`ԇ?S? ?-?7?Α??U??.J??n??t?oJe????w????B?????4jWa?X?O???"'1LgL?8 W4?&??yyzr??Ձkf???LMr:?|Pʒ???B?Zԩ'y????WzCM?CB(͡?+?^??Tz????.gj?????捇?v?{?
                                     /????Lr?^?y|M8(_:eL?v?T?   ?o??u?"S?]??<??COp?/^???a??a?e
                                                                                              ?'
2013/06/24 22:42:34 [009] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63608: connection reset by peer
[HTTPS] First line of body: '???j?@??y???D?%W?Pj?z??z????٢???PB޽?;:X?b?Юg曝?5묑???6X????h?d????'
"?Μ???1?'?p?lWline of body: '?V]??6}^$??Th??U??wI?nۭT?o}??N<!?5vd???V
|??<M?|'????_?N???kP?Q??&E8?Qh????????t1??t?tJq?Øv???s4SKh???1???fR?'_?t<r?*I?;G?Ȟ??4K??
                                                                                        ?b?G-?5V???HD??б?ǚ??!b???B/   ?a???a??ȱ6??D??)"??~??珟G?qܝb2???
                                                 ?????w'

But if I turn off GZip:

package main

import (
    "fmt"
    "github.com/elazarl/goproxy"
    "github.com/elazarl/goproxy/ext/html"
    "net/http"
    "regexp"
    "strings"
)

func main() {
    proxy := goproxy.NewProxyHttpServer()

    proxy.OnRequest(goproxy.ReqHostIs("www.ietf.org:443")).HandleConnect(goproxy.AlwaysMitm)

    proxy.OnRequest(goproxy.UrlMatches(regexp.MustCompile(".*"))).DoFunc(func(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {
        r.Header.Del("Accept-Encoding")
        return r, nil
    })

    proxy.OnResponse(goproxy.DstHostIs("www.ietf.org")).Do(goproxy_html.HandleString(func(s string, ctx *goproxy.ProxyCtx) string {
        fmt.Printf("[HTTP] First line of body: '%s'\n", strings.SplitN(s, "\n", 2)[0])
        return s
    }))

    proxy.OnResponse(goproxy.DstHostIs("www.ietf.org:443")).Do(goproxy_html.HandleString(func(s string, ctx *goproxy.ProxyCtx) string {
        fmt.Printf("[HTTPS] First line of body: '%s'\n", strings.SplitN(s, "\n", 2)[0])
        return s
    }))

    panic(http.ListenAndServe(":8080", proxy))
}

% go run test.go
[HTTP] First line of body: '<!DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">'
2013/06/24 22:48:01 [003] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63735: connection reset by peer
2013/06/24 22:48:01 [005] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63737: connection reset by peer
2013/06/24 22:48:01 [007] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63740: connection reset by peer
2013/06/24 22:48:01 [004] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63736: connection reset by peer
2013/06/24 22:48:01 [008] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63739: connection reset by peer
2013/06/24 22:48:01 [006] WARN: Cannot read TLS request from mitm'd client www.ietf.org:443 read tcp [::1]:63738: connection reset by peer
[HTTPS] First line of body: '<!DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">'

chancancode · 2013-06-25T06:01:13Z

Also, this doesn't work as expected:

    proxy.OnRequest(goproxy.ReqHostIs("www.ietf.org:443")).HandleConnect(goproxy.AlwaysMitm)

    proxy.OnRequest(goproxy.ReqHostIs("www.ietf.org:443")).DoFunc(func(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {
        // This never gets called
        return r, nil
    })

    proxy.OnRequest(goproxy.UrlMatches(regexp.MustCompile(".*"))).DoFunc(func(r *http.Request, ctx *goproxy.ProxyCtx) (*http.Request, *http.Response) {
        // This is the only thing that I could get to work
        return r, nil
    })

chancancode · 2013-06-25T07:11:23Z

There seems to be some other problems with HTTPS. When I return a different string in HandleString in the above example, I get a warning along the lines of "WARN: Cannot write TLS response from mitm'd client http: Request.ContentLength=4063 with Body length 4067". In my real app, this seems to work okay except that the server won't properly close the connection after sending the modified body which causes the browser to wait indefinitely for the content to arrive (or perhaps it's setting the Content-Length header incorrectly).

I hope this will ring some bells, but if not I can investigate further tomorrow and open a new issue for that.

elazarl · 2013-06-26T06:53:50Z

There're still problems with Content-Length. I'm not well right now, so it might take a day or two to solve completely.

elazarl · 2013-06-27T08:19:49Z

Let me know if it works for you. I don't like the current solution, as it costs for every MITM'd requests, even if unmodified, but it should be at least correct.

PS, Thanks a lot for reporting this bugs. These bugs are so stupid it's really embarrassing not to notice them. Keep reporting issues.

goproxy-v1.1: smokescreen compatibility fixes

elazarl closed this as completed in f4f2ef2 Jun 26, 2013

elazarl reopened this Jun 26, 2013

elazarl closed this as completed in 4d91534 Jun 27, 2013

This was referenced Jul 3, 2013

Removing the Content-Length header breaks HEAD requests #11

Closed

Chuncked encoding doesn't seem to work correctly in Chrome #14

Closed

tsgan mentioned this issue Oct 4, 2014

MITM https problem #63

Closed

elazarl mentioned this issue May 30, 2017

CONNECT connections stuck waiting for EOF #209

Closed

HachimanSec mentioned this issue Mar 21, 2019

Outlook phishlet missing correct domain? (goproxy related "Cannot write TLS response body from mitm'd client") kgretzky/evilginx2#248

Closed

bartle-stripe pushed a commit to bartle-stripe/goproxy that referenced this issue Mar 10, 2020

Merge pull request elazarl#9 from stripe/cds/v1.1-smokescreen-compat

7f0347c

goproxy-v1.1: smokescreen compatibility fixes

tobwen mentioned this issue Aug 1, 2024

several warnings: handshake & write TLS body -> broken pipe daijro/hazetunnel#3

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mitm'ed HTTPS responses are not automatically deflated #9

Mitm'ed HTTPS responses are not automatically deflated #9

chancancode commented Jun 25, 2013

elazarl commented Jun 25, 2013

chancancode commented Jun 25, 2013

chancancode commented Jun 25, 2013

chancancode commented Jun 25, 2013

elazarl commented Jun 26, 2013

elazarl commented Jun 27, 2013

Mitm'ed HTTPS responses are not automatically deflated #9

Mitm'ed HTTPS responses are not automatically deflated #9

Comments

chancancode commented Jun 25, 2013

elazarl commented Jun 25, 2013

chancancode commented Jun 25, 2013

chancancode commented Jun 25, 2013

chancancode commented Jun 25, 2013

elazarl commented Jun 26, 2013

elazarl commented Jun 27, 2013