Using utls with http.Transport
#16
Comments
|
I had some success by switching package main
import (
+ gotls "crypto/tls"
"flag"
"fmt"
"io"
@@ -11,6 +12,7 @@ import (
"strings"
utls "github.com/refraction-networking/utls"
+ "golang.org/x/net/http2"
)
func main() {
@@ -51,20 +53,15 @@ func main() {
os.Exit(1)
}
- tr := http.DefaultTransport.(*http.Transport)
+ tr := &http2.Transport{}
if utlsClientHelloID != nil {
- tr.DialContext = nil
- tr.Dial = func(network, addr string) (net.Conn, error) { panic("Dial should not be called") }
- tr.DialTLS = func(network, addr string) (net.Conn, error) {
+ tr.DialTLS = func(network, addr string, cfg *gotls.Config) (net.Conn, error) {
fmt.Printf("DialTLS(%q, %q)\n", network, addr)
- if tr.TLSClientConfig != nil {
- fmt.Printf("warning: ignoring TLSClientConfig %v\n", tr.TLSClientConfig)
- }
conn, err := net.Dial(network, addr)
if err != nil {
return nil, err
}
- uconn := utls.UClient(conn, nil, *utlsClientHelloID)
+ uconn := utls.UClient(conn, &utls.Config{NextProtos: cfg.NextProtos}, *utlsClientHelloID)
colonPos := strings.LastIndex(addr, ":")
if colonPos == -1 {
colonPos = len(addr)These are the results using the URL
Aside from the |
|
Yawning has a commit adding uTLS to obfs4proxy's meek_lite mode: |
|
This is a terrific solution. I'd like to get a wrapper like this into uTLS. |
|
Note that my implementation makes certain assumptions that may not be valid for a more general wrapper (omits some locking, destination host is assumed to be static), but there's comments where I do such things, and altering the behavior should be trivial. |
sergeyfrolov
added
enhancement
Was there ever a proper soloution to use uTLS with the net/http Client? |
|
Comments above describe a proper solution. |
I'm trying to write a wrapper for net.http Client based on your soloution. Am I right in thinking that I would need to execute getTransport() for every new host and do some kind of connection pooling for effeciency? EDIT: I was thinking about this incorrectly regarding connection pooling, The only thing required would be to use something like a map[string]net.Conn so we can handle multiple hosts. Everything seems to be working correctly, after some testing ill submit a PR. |
|
the DialTLS says it's for non-proxied HTTPS requests，But if I want to make requests with transport and proxy，How to make that？ |
There are some code samples here, from the uTLS integration into meek. Adding uTLS was this commit: Adding proxy support was these commits: |
|
The problem (or at least part of it) appears to be here: https://github.com/golang/go/blob/ea1437a8cdf6bb3c2d2447833a5d06dbd75f7ae4/src/net/http/transport.go#L1496 The Since it fails to cast the conn to a What would be ideal is some way to convert a |
Very exciting solution. |
|
Is there any way to use the RoundTripper solution with a Client so you can set Redirect handler and proxy? |
|
This comment is just to report a negative result, an idea I had that turned out not to work. Setting When this issue was crated, go1.11 was current. go1.13 added
The commits that added this field are golang/go@94e7200 and golang/go@2a931ba. The following example program shows that setting demo.go// Demonstration that setting net/http Transport.ForceAttemptHTTP2 to true does
// not suffice to enable HTTP/2 support when its dialer returns a *utls.Conn,
// rather than a *tls.Conn.
package main
import (
"context"
"flag"
"fmt"
"net"
"net/http"
"os"
utls "github.com/refraction-networking/utls"
)
var utlsClientHelloID = &utls.HelloFirefox_65
// utlsDialContext connects to the given network address and initiates a TLS
// handshake with the provided ClientHelloID, and returns the resulting TLS
// connection.
func utlsDialContext(ctx context.Context, network, addr string, config *utls.Config, id *utls.ClientHelloID) (*utls.UConn, error) {
// Set the SNI from addr, if not already set.
if config == nil {
config = &utls.Config{}
}
if config.ServerName == "" {
config = config.Clone()
host, _, err := net.SplitHostPort(addr)
if err != nil {
return nil, err
}
config.ServerName = host
}
dialer := &net.Dialer{}
conn, err := dialer.DialContext(ctx, network, addr)
if err != nil {
return nil, err
}
uconn := utls.UClient(conn, config, *id)
// Manually remove the SNI if it contains an IP address.
// https://github.com/refraction-networking/utls/issues/96
if net.ParseIP(config.ServerName) != nil {
err := uconn.RemoveSNIExtension()
if err != nil {
uconn.Close()
return nil, err
}
}
// We must call Handshake before returning, or else the UConn may not
// actually use the selected ClientHelloID. It depends on whether a Read
// or a Write happens first. If a Read happens first, the connection
// will use the normal crypto/tls fingerprint. If a Write happens first,
// it will use the selected fingerprint as expected.
// https://github.com/refraction-networking/utls/issues/75
err = uconn.Handshake()
if err != nil {
uconn.Close()
return nil, err
}
return uconn, nil
}
func fetch(url string) error {
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.DialTLSContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
return utlsDialContext(ctx, network, addr, nil, utlsClientHelloID)
}
transport.ForceAttemptHTTP2 = true
client := http.Client{
Transport: transport,
}
resp, err := client.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
fmt.Println(resp.Proto)
return nil
}
func main() {
flag.Parse()
if flag.NArg() != 1 {
fmt.Fprintf(os.Stderr, "usage: %s URL\n", os.Args[0])
os.Exit(1)
}
err := fetch(flag.Arg(0))
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
|
|
Anyone solved this issue ? |
There's an [open thread](refraction-networking/utls#16) in uTLS about dial support for http/2 servers. The tldr is that by returning a utls.Connection (versus a tls.Connection ) in the dial function it fails the test specified in [root handler](https://github.com/golang/go/blob/ea1437a8cdf6bb3c2d2447833a5d06dbd75f7ae4/src/net/http/transport.go#L1496) and therefore doesn't keep track of the connection state. This results in the regular connection attempting to request http/2 connection through an http/1 protocol. We work around this through meek's re-implementation of the roundtrip handler that dynamically routes between http/1 and http/2.
|
you can implement your own package fetch
import (
"bufio"
"fmt"
"net"
"net/http"
"sync"
utls "github.com/refraction-networking/utls"
"golang.org/x/net/http2"
)
func NewBypassJA3Transport(helloID utls.ClientHelloID) *BypassJA3Transport {
return &BypassJA3Transport{clientHello: helloID}
}
type BypassJA3Transport struct {
tr1 http.Transport
tr2 http2.Transport
mu sync.RWMutex
clientHello utls.ClientHelloID
}
func (b *BypassJA3Transport) RoundTrip(req *http.Request) (*http.Response, error) {
switch req.URL.Scheme {
case "https":
return b.httpsRoundTrip(req)
case "http":
return b.tr1.RoundTrip(req)
default:
return nil, fmt.Errorf("unsupported scheme: %s", req.URL.Scheme)
}
}
func (b *BypassJA3Transport) httpsRoundTrip(req *http.Request) (*http.Response, error) {
port := req.URL.Port()
if port == "" {
port = "443"
}
conn, err := net.Dial("tcp", fmt.Sprintf("%s:%s", req.URL.Host, port))
if err != nil {
return nil, fmt.Errorf("tcp net dial fail: %w", err)
}
defer conn.Close() // nolint
tlsConn, err := b.tlsConnect(conn, req)
if err != nil {
return nil, fmt.Errorf("tls connect fail: %w", err)
}
httpVersion := tlsConn.ConnectionState().NegotiatedProtocol
switch httpVersion {
case "h2":
conn, err := b.tr2.NewClientConn(tlsConn)
if err != nil {
return nil, fmt.Errorf("create http2 client with connection fail: %w", err)
}
defer conn.Close() // nolint
return conn.RoundTrip(req)
case "http/1.1", "":
err := req.Write(tlsConn)
if err != nil {
return nil, fmt.Errorf("write http1 tls connection fail: %w", err)
}
return http.ReadResponse(bufio.NewReader(tlsConn), req)
default:
return nil, fmt.Errorf("unsuported http version: %s", httpVersion)
}
}
func (b *BypassJA3Transport) getTLSConfig(req *http.Request) *utls.Config {
return &utls.Config{
ServerName: req.URL.Host,
InsecureSkipVerify: true,
}
}
func (b *BypassJA3Transport) tlsConnect(conn net.Conn, req *http.Request) (*utls.UConn, error) {
b.mu.RLock()
tlsConn := utls.UClient(conn, b.getTLSConfig(req), b.clientHello)
b.mu.RUnlock()
if err := tlsConn.Handshake(); err != nil {
return nil, fmt.Errorf("tls handshake fail: %w", err)
}
return tlsConn, nil
}
func (b *BypassJA3Transport) SetClientHello(hello utls.ClientHelloID) {
b.mu.Lock()
defer b.mu.Unlock()
b.clientHello = hello
}and set Tranport is enough: var client = &http.Client{
Timeout: time.Second * 30,
Transport: NewBypassJA3Transport(utls.HelloChrome_102),
} |
|
if it helps, I came up with the below implementation of package tls
import (
"bufio"
"github.com/refraction-networking/utls"
"net"
"net/http"
)
type Transport struct {
Conn *tls.UConn
Spec tls.ClientHelloSpec
}
func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error) {
config := tls.Config{ServerName: req.URL.Host}
conn, err := net.Dial("tcp", req.URL.Host+":443")
if err != nil {
return nil, err
}
t.Conn = tls.UClient(conn, &config, tls.HelloCustom)
if err := t.Conn.ApplyPreset(&t.Spec); err != nil {
return nil, err
}
if err := req.Write(t.Conn); err != nil {
return nil, err
}
return http.ReadResponse(bufio.NewReader(t.Conn), req)
} |
|
Don't know if anyone is still concerned with this stale issue, but if the purpose is simply for building HTTP Clients with uTLS as the tls stack, imroc/req seemed to be a good choice. See imroc/req#198 for multiple different ways you may use uTLS with imroc/req. Closing this for now, but if issue not considered resolved we may reopen it. |
and others
I'm using commit a89e7e6. The examples I have found of using utls with HTTPS all make a single request on a single connection, then throw the connection away. For example,
httpGetOverConnin examples.go.I'm trying to use utls with
http.Transport, to take advantage of persistent connections and reasonable default timeouts. To do this, I'm hooking into theDialTLScallback. There is a problem when using a utls fingerprint that includesh2in ALPN and a server that supports HTTP/2. The server switches to HTTP/2 mode, but the client stays in HTTP/1.1 mode, because net/http disables automatic HTTP/2 support wheneverDialTLSis set. The end result is an HTTP/1.1 client speaking to an HTTP/2 server; i.e, a similar problem as what was reported in golang/go#14275 (comment). The error message differs depending on the fingerprint:HelloFirefox_63net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x00@\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01\x00\x00\x1e\a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01http2_handshake_failed"HelloChrome_70local error: tls: unexpected messageHelloIOS_11_12019/01/11 14:48:56 Unsolicited response received on idle HTTP channel starting with "\x00\x00\x12\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00d\x00\x04\x00\x10\x00\x00\x00\x06\x00\x00@\x00\x00\x00\x04\b\x00\x00\x00\x00\x00\x00\x0f\x00\x01"; err=<nil>readLoopPeekFailLocked: <nil>
I get the same results even if I pre-configure the
http.Transportwith HTTP/2 support by callinghttp2.ConfigureTransport(tr).I wrote a test program to reproduce these results. It takes a
-utlsoption to select a utls client hello ID, and a-callhandshakeoption to control whether to callUConn.HandshakewithinDialTLS, or allow it to be called implicitly by the nextReadorWrite. I included the latter option because I found that not callingUConn.HandshakeinsideDialTLSavoids the HTTP version mismatch; however it also results in a client hello that lacks ALPN and differs from the requested one in other ways, so it's not an adequate workaround.Click to expand program
Sample usage:
The output of the program appears in the following table. Things to notice:
DialTLSwithHelloGolangproduces a fingerprint that is different from usinghttp.TransportwithoutDialTLSset.HelloFirefox_63,HelloChrome_70, andHelloIOS_11_1all provide a usable connection (but with an incorrect fingerprint), as long as you don't callUConn.Handshakebefore returning fromDialTLS.HelloFirefox_63,HelloChrome_70, andHelloIOS_11_1all give the correct fingerprint, but fail with an HTTP version mismatch, whenUConn.Handshakeis called insideDialTLS.Handshake?-utls HelloGolang-utls HelloGolang-callhandshake-utls HelloFirefox_63-utls HelloFirefox_63-callhandshakemalformed HTTP response(HTTP/1.1 client, HTTP/2 server)-utls HelloChrome_70-utls HelloChrome_70-callhandshakelocal error: tls: unexpected message-utls HelloIOS_11_1-utls HelloIOS_11_1-callhandshakereadLoopPeekFailLocked: <nil>(HTTP/1.1 client, HTTP/2 server)Is there a way to accomplish what I am trying to do?
The text was updated successfully, but these errors were encountered: