Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using utls with `http.Transport` #16

Open
AxbB36 opened this issue Jan 11, 2019 · 7 comments

Comments

Projects
None yet
4 participants
@AxbB36
Copy link

commented Jan 11, 2019

I'm using commit a89e7e6. The examples I have found of using utls with HTTPS all make a single request on a single connection, then throw the connection away. For example, httpGetOverConn in examples.go.

I'm trying to use utls with http.Transport, to take advantage of persistent connections and reasonable default timeouts. To do this, I'm hooking into the DialTLS callback. There is a problem when using a utls fingerprint that includes h2 in ALPN and a server that supports HTTP/2. The server switches to HTTP/2 mode, but the client stays in HTTP/1.1 mode, because net/http disables automatic HTTP/2 support whenever DialTLS is set. The end result is an HTTP/1.1 client speaking to an HTTP/2 server; i.e, a similar problem as what was reported in golang/go#14275 (comment). The error message differs depending on the fingerprint:

HelloFirefox_63
net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x00@\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01\x00\x00\x1e\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01http2_handshake_failed"
HelloChrome_70
local error: tls: unexpected message
HelloIOS_11_1
2019/01/11 14:48:56 Unsolicited response received on idle HTTP channel starting with "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x00@\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01"; err=<nil>
readLoopPeekFailLocked: <nil>

I get the same results even if I pre-configure the http.Transport with HTTP/2 support by calling http2.ConfigureTransport(tr).

I wrote a test program to reproduce these results. It takes a -utls option to select a utls client hello ID, and a -callhandshake option to control whether to call UConn.Handshake within DialTLS, or allow it to be called implicitly by the next Read or Write. I included the latter option because I found that not calling UConn.Handshake inside DialTLS avoids the HTTP version mismatch; however it also results in a client hello that lacks ALPN and differs from the requested one in other ways, so it's not an adequate workaround.

Click to expand program
package main

import (
	"flag"
	"fmt"
	"io"
	"io/ioutil"
	"net"
	"net/http"
	"os"
	"strings"

	utls "github.com/refraction-networking/utls"
)

func main() {
	utlsClientHelloIDName := flag.String("utls", "", "use utls with the given ClientHelloID (e.g. HelloGolang)")
	callHandshake := flag.Bool("callhandshake", false, "call UConn.Handshake inside DialTLS")
	flag.Parse()

	if *callHandshake && *utlsClientHelloIDName == "" {
		fmt.Fprintf(os.Stderr, "error: -callhandshake only makes sense with -utls\n")
		os.Exit(1)
	}

	if flag.NArg() != 1 {
		fmt.Fprintf(os.Stderr, "error: need a URL\n")
		os.Exit(1)
	}
	url := flag.Arg(0)

	utlsClientHelloID, ok := map[string]*utls.ClientHelloID{
		"":                      nil,
		"HelloGolang":           &utls.HelloGolang,
		"HelloRandomized":       &utls.HelloRandomized,
		"HelloRandomizedALPN":   &utls.HelloRandomizedALPN,
		"HelloRandomizedNoALPN": &utls.HelloRandomizedNoALPN,
		"HelloFirefox_Auto":     &utls.HelloFirefox_Auto,
		"HelloFirefox_55":       &utls.HelloFirefox_55,
		"HelloFirefox_56":       &utls.HelloFirefox_56,
		"HelloFirefox_63":       &utls.HelloFirefox_63,
		"HelloChrome_Auto":      &utls.HelloChrome_Auto,
		"HelloChrome_58":        &utls.HelloChrome_58,
		"HelloChrome_62":        &utls.HelloChrome_62,
		"HelloChrome_70":        &utls.HelloChrome_70,
		"HelloIOS_Auto":         &utls.HelloIOS_Auto,
		"HelloIOS_11_1":         &utls.HelloIOS_11_1,
	}[*utlsClientHelloIDName]
	if !ok {
		fmt.Fprintf(os.Stderr, "unknown client hello ID %q\n", *utlsClientHelloIDName)
		os.Exit(1)
	}

	tr := http.DefaultTransport.(*http.Transport)
	if utlsClientHelloID != nil {
		tr.DialContext = nil
		tr.Dial = func(network, addr string) (net.Conn, error) { panic("Dial should not be called") }
		tr.DialTLS = func(network, addr string) (net.Conn, error) {
			fmt.Printf("DialTLS(%q, %q)\n", network, addr)
			if tr.TLSClientConfig != nil {
				fmt.Printf("warning: ignoring TLSClientConfig %v\n", tr.TLSClientConfig)
			}
			conn, err := net.Dial(network, addr)
			if err != nil {
				return nil, err
			}
			uconn := utls.UClient(conn, nil, *utlsClientHelloID)
			colonPos := strings.LastIndex(addr, ":")
			if colonPos == -1 {
				colonPos = len(addr)
			}
			uconn.SetSNI(addr[:colonPos])
			if *callHandshake {
				err = uconn.Handshake()
			}
			return uconn, err
		}
	}

	for i := 0; i < 4; i++ {
		resp, err := get(tr, url)
		if err != nil {
			fmt.Printf("%2d err %v\n", i, err)
		} else {
			fmt.Printf("%2d %s %s\n", i, resp.Proto, resp.Status)
		}
	}
}

func get(rt http.RoundTripper, url string) (*http.Response, error) {
	req, err := http.NewRequest("GET", url, nil)
	if err != nil {
		return nil, err
	}
	resp, err := rt.RoundTrip(req)
	if err != nil {
		return nil, err
	}
	// Read and close the body to enable connection reuse with HTTP/1.1.
	_, err = io.Copy(ioutil.Discard, resp.Body)
	if err != nil {
		return nil, err
	}
	err = resp.Body.Close()
	if err != nil {
		return nil, err
	}
	return resp, nil
}

Sample usage:

test -utls HelloFirefox_63 -callhandshake https://golang.org/robots.txt

The output of the program appears in the following table. Things to notice:

  • DialTLS with HelloGolang produces a fingerprint that is different from using http.Transport without DialTLS set.
  • HelloFirefox_63, HelloChrome_70, and HelloIOS_11_1 all provide a usable connection (but with an incorrect fingerprint), as long as you don't call UConn.Handshake before returning from DialTLS.
  • HelloFirefox_63, HelloChrome_70, and HelloIOS_11_1 all give the correct fingerprint, but fail with an HTTP version mismatch, when UConn.Handshake is called inside DialTLS.
Client Hello ID call Handshake? client ALPN result
none N/A [h2, http/1.1] ok HTTP/2
-utls HelloGolang none ok HTTP/1.1
-utls HelloGolang -callhandshake none ok HTTP/1.1
-utls HelloFirefox_63 none ok HTTP/1.1
-utls HelloFirefox_63 -callhandshake [h2, http/1.1] malformed HTTP response (HTTP/1.1 client, HTTP/2 server)
-utls HelloChrome_70 none ok HTTP/1.1
-utls HelloChrome_70 -callhandshake [h2, http/1.1] local error: tls: unexpected message
-utls HelloIOS_11_1 none ok HTTP/1.1
-utls HelloIOS_11_1 -callhandshake [h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1] readLoopPeekFailLocked: <nil> (HTTP/1.1 client, HTTP/2 server)

Is there a way to accomplish what I am trying to do?

@AxbB36

This comment has been minimized.

Copy link
Author

commented Jan 12, 2019

I had some success by switching http2.Transport for http.Transport:

 package main
 
 import (
+	gotls "crypto/tls"
 	"flag"
 	"fmt"
 	"io"
@@ -11,6 +12,7 @@ import (
 	"strings"
 
 	utls "github.com/refraction-networking/utls"
+	"golang.org/x/net/http2"
 )
 
 func main() {
@@ -51,20 +53,15 @@ func main() {
 		os.Exit(1)
 	}
 
-	tr := http.DefaultTransport.(*http.Transport)
+	tr := &http2.Transport{}
 	if utlsClientHelloID != nil {
-		tr.DialContext = nil
-		tr.Dial = func(network, addr string) (net.Conn, error) { panic("Dial should not be called") }
-		tr.DialTLS = func(network, addr string) (net.Conn, error) {
+		tr.DialTLS = func(network, addr string, cfg *gotls.Config) (net.Conn, error) {
 			fmt.Printf("DialTLS(%q, %q)\n", network, addr)
-			if tr.TLSClientConfig != nil {
-				fmt.Printf("warning: ignoring TLSClientConfig %v\n", tr.TLSClientConfig)
-			}
 			conn, err := net.Dial(network, addr)
 			if err != nil {
 				return nil, err
 			}
-			uconn := utls.UClient(conn, nil, *utlsClientHelloID)
+			uconn := utls.UClient(conn, &utls.Config{NextProtos: cfg.NextProtos}, *utlsClientHelloID)
 			colonPos := strings.LastIndex(addr, ":")
 			if colonPos == -1 {
 				colonPos = len(addr)

These are the results using the URL https://golang.org/robots.txt. Notice:

  • The output is the same whether UConn.Handshake is called within DialTLS or not.
  • The HelloChrome_70 fingerprint doesn't work.
Client Hello ID call Handshake? client ALPN result
none N/A [h2] ok HTTP/2
-utls HelloGolang [h2] ok HTTP/2
-utls HelloGolang -callhandshake [h2] ok HTTP/2
-utls HelloFirefox_63 [h2, http/1.1] ok HTTP/2
-utls HelloFirefox_63 -callhandshake [h2, http/1.1] ok HTTP/2
-utls HelloChrome_70 [h2, http/1.1] local error: tls: unexpected message
-utls HelloChrome_70 -callhandshake [h2, http/1.1] local error: tls: unexpected message
-utls HelloIOS_11_1 [h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1] ok HTTP/2
-utls HelloIOS_11_1 -callhandshake [h2, h2-16, h2-15, h2-14, spdy/3.1, spdy/3, http/1.1] ok HTTP/2

Aside from the HelloChrome_70 failure, this is basically what I want. The only problem is that it doesn't work against non-HTTP/2 servers, whether utls is used or not:

$ test https://apache.org/
 0 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
 1 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
 2 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
 3 err http2: unexpected ALPN protocol "http/1.1"; want "h2"
$ test -utls HelloGolang -callhandshake https://apache.org/
DialTLS("tcp", "apache.org:443")
 0 err unexpected EOF
DialTLS("tcp", "apache.org:443")
 1 err unexpected EOF
DialTLS("tcp", "apache.org:443")
 2 err unexpected EOF
DialTLS("tcp", "apache.org:443")
 3 err unexpected EOF
@AxbB36

This comment has been minimized.

Copy link
Author

commented Jan 21, 2019

Yawning has a commit adding uTLS to obfs4proxy's meek_lite mode:
https://gitlab.com/yawning/obfs4/commit/4d453dab2120082b00bf6e63ab4aaeeda6b8d8a3
It uses an I idea I didn't think of. Instead of setting DialTLS on an http.Transport and using it directly, it uses a wrapper http.RoundTripper. On the first dial, the wrapper initiates the UClient connection, then inspects conn.ConnectionState().NegotiatedProtocol. Depending on the negotiated protocol, it creates internally either an http.Transport or http2.Transport. Future dials are passed through to the internal transport directly.

@sergeyfrolov

This comment has been minimized.

Copy link
Member

commented Jan 21, 2019

This is a terrific solution. I'd like to get a wrapper like this into uTLS.

@Yawning

This comment has been minimized.

Copy link

commented Jan 21, 2019

Note that my implementation makes certain assumptions that may not be valid for a more general wrapper (omits some locking, destination host is assumed to be static), but there's comments where I do such things, and altering the behavior should be trivial.

@zanitee

This comment has been minimized.

Copy link

commented May 1, 2019

This is a terrific solution. I'd like to get a wrapper like this into uTLS.

Was there ever a proper soloution to use uTLS with the net/http Client?

@sergeyfrolov

This comment has been minimized.

Copy link
Member

commented May 1, 2019

Comments above describe a proper solution.
This wrapper had not been implemented in uTLS yet.
image

@zanitee

This comment has been minimized.

Copy link

commented May 2, 2019

Note that my implementation makes certain assumptions that may not be valid for a more general wrapper (omits some locking, destination host is assumed to be static), but there's comments where I do such things, and altering the behavior should be trivial.

I'm trying to write a wrapper for net.http Client based on your soloution. Am I right in thinking that I would need to execute getTransport() for every new host and do some kind of connection pooling for effeciency?

EDIT: I was thinking about this incorrectly regarding connection pooling, The only thing required would be to use something like a map[string]net.Conn so we can handle multiple hosts. Everything seems to be working correctly, after some testing ill submit a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.