New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Transport doesn't support NTLM challenge authentication #20053

Open
chen-keinan opened this Issue Apr 20, 2017 · 11 comments

Comments

Projects
None yet
4 participants
@chen-keinan

chen-keinan commented Apr 20, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

Go 1.8

What operating system and processor architecture are you using (go env)?

  OSX darwin-amd64

What did you do?

I have send an https request to a proxy (ntlm) below request and initial response
(via wireshark)

Request:
CONNECT www.endpoint.com:443 HTTP/1.1
Host: www.endpoint.com:443
User-Agent: Go-http-client/1.1
Location: https://www.endpoint.com
Proxy-Authorization: NTLM TlRMTVNTUAABAAAAB4IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAAAAAAAMAA=
------------------------------------------------------------------------------------------
Response
HTTP/1.1 407 Proxy Authentication Required
Server: FreeProxy/4.50
Date: Thu, 20 Apr 2017 15:20:10 GMT
Content-Type: text/html
Transfer-Encoding: Chunked
Proxy-Authenticate: NTLM
TlRMTVNTUAACAAAADAAMADgAAAAFgoECloLVra5EaVAAAAAAAAAAA
A9KAEYAUgBPAEcAMAACAAwASgBGAFIATwBHADAAAQAOAFcASQBOA
ZgByAG8AZwAuAGwAbwBjAGEAbAADACYAdwBpAG4AMgAwADEAMgAu
wAbwBjAGEAbAAFABYAagBmAHIAbwBnAC4AbABvAGMAYQBsAAcACAD
Proxy-Connection: Keep-Alive
------------------------------------------------------------------------------------------------
The response above never reach the client, on transport.dialConn the response return status code 407 for challenge , because the response code != 200 the persist connection become nil

  -------------------------------------------------------------------------------------------
	br := bufio.NewReader(conn)
	resp, err := ReadResponse(br, connectReq) // resp.StatusCode =407
	if err != nil {
		conn.Close()
		return nil, err
	}
	if resp.StatusCode != 200 {
		f := strings.SplitN(resp.Status, " ", 2)
		conn.Close()
		return nil, errors.New(f[1]) // persist connection become nil 
	}

since the persist connection return nil then request is cancelled and response return as nil
with error Proxy Authentication Required
see --> transport.RoundTrip

 --------------------------------------------------------------------------------------------
     pconn, err := t.getConn(treq, cm) // pconn = nil
	if err != nil {
		t.setReqCanceler(req, nil)
		req.closeBody()
		return nil, err
	}
-------------------------------------------------------------------------------------------------

What did you expect to see?

I expect the response to return is it send from the proxy with status code 407

What did you see instead?

I got nil response with error: Proxy Authentication Required

Note: if I use http instead of https it works OK

This issue is blocking us from developing support to NTLM Proxy , as requests https endpoint do not return challenge from proxy

@chen-keinan chen-keinan changed the title from https request via proxy which return 407 return nil response to https request via proxy return nil response Apr 20, 2017

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Apr 20, 2017

Member

I believe this is all working as intended.

If your proxy returns an error, we don't want to return its HTTP response to the user, as that would imply the origin server replied with that.

You can use ProxyConnectHeader to authenticate to your proxy.

Let me know if I misunderstand something.

Member

bradfitz commented Apr 20, 2017

I believe this is all working as intended.

If your proxy returns an error, we don't want to return its HTTP response to the user, as that would imply the origin server replied with that.

You can use ProxyConnectHeader to authenticate to your proxy.

Let me know if I misunderstand something.

@bradfitz bradfitz closed this Apr 20, 2017

@chen-keinan

This comment has been minimized.

Show comment
Hide comment
@chen-keinan

chen-keinan Apr 20, 2017

I did used ProxyConnectHeader to send the Proxy-Authorization which is OK , at that point the CONNECT NEGOTIATE started , then proxy return 407 with Challenge header which I do not received in the client due to issue describe above.
407 is good , it is part of the NTLM proto it will enable me to continue with negotiation , same as its done with http.

since the response come as nil I cannot process the response , if the response would return with 407 and challenge (as proxy send it) it will help me, on the client side to decide on how to continue

chen-keinan commented Apr 20, 2017

I did used ProxyConnectHeader to send the Proxy-Authorization which is OK , at that point the CONNECT NEGOTIATE started , then proxy return 407 with Challenge header which I do not received in the client due to issue describe above.
407 is good , it is part of the NTLM proto it will enable me to continue with negotiation , same as its done with http.

since the response come as nil I cannot process the response , if the response would return with 407 and challenge (as proxy send it) it will help me, on the client side to decide on how to continue

@bradfitz bradfitz reopened this Apr 20, 2017

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Apr 20, 2017

Member

I see. It's true we don't support authentication that takes multiple rounds.

Member

bradfitz commented Apr 20, 2017

I see. It's true we don't support authentication that takes multiple rounds.

@bradfitz bradfitz added this to the Unplanned milestone Apr 20, 2017

@bradfitz bradfitz changed the title from https request via proxy return nil response to net/http: Transport doesn't support NTLM challenge authentication Apr 20, 2017

@chen-keinan

This comment has been minimized.

Show comment
Hide comment
@chen-keinan

chen-keinan Apr 20, 2017

the same NTLM negotiation works in http , why is https different ?

chen-keinan commented Apr 20, 2017

the same NTLM negotiation works in http , why is https different ?

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Apr 20, 2017

Member

Because HTTPS does CONNECT and authenticates to that to give your channel for future requests.

With HTTP you're kinda getting lucky and happen to be using the same TCP connection I suppose, but there was never explicit design or support for what you're trying to do.

Member

bradfitz commented Apr 20, 2017

Because HTTPS does CONNECT and authenticates to that to give your channel for future requests.

With HTTP you're kinda getting lucky and happen to be using the same TCP connection I suppose, but there was never explicit design or support for what you're trying to do.

@chen-keinan

This comment has been minimized.

Show comment
Hide comment
@chen-keinan

chen-keinan Apr 20, 2017

do you have any suggested workaround for this issue in the meantime ?

chen-keinan commented Apr 20, 2017

do you have any suggested workaround for this issue in the meantime ?

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Apr 20, 2017

Member

Implement Transport.DialContext (but leave Transport.Proxy nil) and do your own CONNECT setup before giving the Conn back to the http package?

Member

bradfitz commented Apr 20, 2017

Implement Transport.DialContext (but leave Transport.Proxy nil) and do your own CONNECT setup before giving the Conn back to the http package?

@gogolok

This comment has been minimized.

Show comment
Hide comment
@gogolok

gogolok Jul 10, 2017

hey @chenkjfrog

I've built a modified Go version that supports NTLM proxies.
I'm using it to build a Cloud Foundry CLI with NTLM support.
It's WIP and I'm not sure what solution someone should target to support this in the Go base/master in the future but I hope it can somehow help you :-)

https://gist.github.com/gogolok/018443687392ea4682bd82ac8712b363

gogolok commented Jul 10, 2017

hey @chenkjfrog

I've built a modified Go version that supports NTLM proxies.
I'm using it to build a Cloud Foundry CLI with NTLM support.
It's WIP and I'm not sure what solution someone should target to support this in the Go base/master in the future but I hope it can somehow help you :-)

https://gist.github.com/gogolok/018443687392ea4682bd82ac8712b363

@gogolok

This comment has been minimized.

Show comment
Hide comment
@gogolok

gogolok Sep 8, 2017

I'm currently working on extending http standard library to support authentication that takes multiple rounds.

Hopefully I can present something in the near future.

gogolok commented Sep 8, 2017

I'm currently working on extending http standard library to support authentication that takes multiple rounds.

Hopefully I can present something in the near future.

@bhendo

This comment has been minimized.

Show comment
Hide comment
@bhendo

bhendo Sep 25, 2017

@gogolok Thanks for this!

I want to bring one thing to your attention.

I've been doing some testing with your patch and have found one scenario that does not work.
When a request is made with the http scheme through an NTLM proxy that also intercepts the request, the proxy does not forward the request to the end server. My understanding is that the CONNECT method is meant for tunneling https requests only.

I think the reason the patch works for proxies that don't intercept is because the proxy "blindly" establishes a connection from the client to the server. But when a proxy is in intercept mode it actually maintains two connections: client<->proxy & proxy<->server

I captured some packets with WireShark to see how chrome handles http requests through an NTLM proxy. It seems to use the original request, method (e.g. GET, POST, etc), and body for each part of the handshake.

After a quick review of transport.go I think the NTLM handshake specifically for http requests might need to take place in RoundTrip rather than dialConn

I will admit this all is a bit out of my wheelhouse, so take my last statement with a grain of salt :)

bhendo commented Sep 25, 2017

@gogolok Thanks for this!

I want to bring one thing to your attention.

I've been doing some testing with your patch and have found one scenario that does not work.
When a request is made with the http scheme through an NTLM proxy that also intercepts the request, the proxy does not forward the request to the end server. My understanding is that the CONNECT method is meant for tunneling https requests only.

I think the reason the patch works for proxies that don't intercept is because the proxy "blindly" establishes a connection from the client to the server. But when a proxy is in intercept mode it actually maintains two connections: client<->proxy & proxy<->server

I captured some packets with WireShark to see how chrome handles http requests through an NTLM proxy. It seems to use the original request, method (e.g. GET, POST, etc), and body for each part of the handshake.

After a quick review of transport.go I think the NTLM handshake specifically for http requests might need to take place in RoundTrip rather than dialConn

I will admit this all is a bit out of my wheelhouse, so take my last statement with a grain of salt :)

@gogolok

This comment has been minimized.

Show comment
Hide comment
@gogolok

gogolok Oct 12, 2017

@bhendo Thanks for your feedback. I'm currently testing and extending my new implementation and will try to consider the intercept mode. I might come back to you then :-)

gogolok commented Oct 12, 2017

@bhendo Thanks for your feedback. I'm currently testing and extending my new implementation and will try to consider the intercept mode. I might come back to you then :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment