proxy-ntlm requires Proxy-Connection: keep-alive

[fajan]

I did this

In version 7.48.0 changelog: "Proxy-Connection: stop sending this header by default"
Source issue #633

"NTLM authentication cannot be used if an intervening proxy does not support keep-alive connections." (msdn)

Without the 'Proxy-Connection' header, --proxy-ntlm fails.

Maybe related issue: #876

curl 7.50.1 command:

$ curl --proxy-ntlm --proxy-user "MyDomain\ntlmtest:test" --proxy proxy.estest.intra:3128 -v https://example.com/file
*   Trying 101.102.103.104...
* Connected to proxy.estest.intra (101.102.103.104) port 3128 (#0)
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain\ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNTUAA...AAAAAAAA=
> User-Agent: curl/7.50.1
>
< HTTP/1.0 407 Proxy Authentication Required
< Server: squid/2.7.STABLE5
< Date: Thu, 11 Aug 2016 08:17:15 GMT
< Content-Type: text/html
< Content-Length: 1330
< X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
< Proxy-Authenticate: Basic realm="Squid proxy-caching web server"
< X-Cache: MISS from intra.my.domain.com
< X-Cache-Lookup: NONE from intra.my.domain.com:3128
< Via: 1.0 intra.my.domain.com:3128 (squid/2.7.STABLE5)
< Connection: close
<
* Ignore 1330 bytes of response-body
* Received HTTP code 407 from proxy after CONNECT
* Closing connection 0
curl: (56) Received HTTP code 407 from proxy after CONNECT

I expected the following

same command with 7.47.1

$c:/programs/curl/7.47.1/curl --proxy-ntlm --proxy-user "MyDomain\ntlmtest:test" --proxy proxy.estest.intra:3128 -v https://example.com/file
*   Trying 101.102.103.104...
* Connected to proxy.estest.intra (101.102.103.104) port 3128 (#0)
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain\ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNTUAA...AAAAAAAA=
> User-Agent: curl/7.47.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 407 Proxy Authentication Required
< Server: squid/2.7.STABLE5
< Date: Thu, 11 Aug 2016 09:18:33 GMT
< Content-Type: text/html
< Content-Length: 1330
< X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
< Proxy-Authenticate: NTLM TlRMTVNTUAA.....AuAGgAdQAAAAAA
< X-Cache: MISS from intra.my.domain.com
< X-Cache-Lookup: NONE from intra.my.domain.com:3128
< Via: 1.0 intra.my.domain.com:3128 (squid/2.7.STABLE5)
< Connection: keep-alive
< Proxy-Connection: keep-alive
<
* Ignore 1330 bytes of response-body
* TUNNEL_STATE switched to: 0
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain\ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNTUAA........1GRUpMMTM=
> User-Agent: curl/7.47.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
....

command on 7.50.1. with --proxy-header proxy-connection:keep-alive extra options

$ curl --proxy-ntlm --proxy-user "MyDomain\ntlmtest:test" --proxy proxy.estest.intra:3128 --proxy-header proxy-connection:keep-alive -v https://example.com/file
*   Trying 101.102.103.104...
* Connected to proxy.estest.intra (101.102.103.104) port 3128 (#0)
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain\ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNTUAA...AAAAAAA=
> User-Agent: curl/7.50.1
> proxy-connection:keep-alive
>
< HTTP/1.0 407 Proxy Authentication Required
< Server: squid/2.7.STABLE5
< Date: Thu, 11 Aug 2016 08:18:55 GMT
< Content-Type: text/html
< Content-Length: 1330
< X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
< Proxy-Authenticate: NTLM TlRMTVNTUAA.....AuAGgAdQAAAAAA
< X-Cache: MISS from intra.my.domain.com
< X-Cache-Lookup: NONE from intra.my.domain.com:3128
< Via: 1.0 intra.my.domain.com:3128 (squid/2.7.STABLE5)
< Connection: keep-alive
< Proxy-Connection: keep-alive
<
* Ignore 1330 bytes of response-body
* TUNNEL_STATE switched to: 0
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain\ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNTUAA........1GRUpMMTM=
> User-Agent: curl/7.50.1
> proxy-connection:keep-alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request
....

curl/libcurl version

$ curl -V
curl 7.50.1 (x86_64-pc-win32) libcurl/7.50.1 OpenSSL/1.0.2g
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM SSL

$ c:/programs/curl/7.47.1/curl -V
curl 7.47.1 (x86_64-pc-win32) libcurl/7.47.1 OpenSSL/1.0.2g
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM SSL

The same with ENABLE_SSPI=yes versions of above.

[bagder]

So if you revert 113f04e from git master, it works again?

[fajan]

Yes.

$ curl --proxy-ntlm --proxy proxy.estest.intra:3128 --proxy-user MyDomain\ntlmtest:test -v https://example.com/file
*   Trying 101.102.103.104...
* TCP_NODELAY set
* Connected to proxy.estest.intra (101.102.103.104) port 3128 (#0)
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain\ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNTU...AAAAAAAA=
> User-Agent: curl/7.50.2-DEV
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 407 Proxy Authentication Required
< Server: squid/2.7.STABLE5
< Date: Fri, 12 Aug 2016 06:06:00 GMT
< Content-Type: text/html
< Content-Length: 1333
< X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
< Proxy-Authenticate: NTLM TlRMTVNTUAA.....QAAAAAA
< X-Cache: MISS from intra.my.domain.com
< X-Cache-Lookup: NONE from intra.my.domain.com:3128
< Via: 1.0 intra.my.domain.com:3128 (squid/2.7.STABLE5)
< Connection: keep-alive
< Proxy-Connection: keep-alive
<
* Ignore 1333 bytes of response-body
* TUNNEL_STATE switched to: 0
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain\ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNTU....SS1GRUpMMTM=
> User-Agent: curl/7.50.2-DEV
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request

[bagder]

Any idea what proxy/version this is that's causing this trouble? As mentioned in the issue, the header is deprecated and the advice is not to use...

[fajan]

Well, ntlm isn't today's tech, but still used.
I would suggest to only set 'Proxy-Connection' header if the auth mode is ntlm.
Systems that support this auth won't refuse the header especially if paired with 'Proxy-Authorization: NTLM ...'.

[bagder]

Well, this isn't strictly because of NTLM as NTLM over proxy works perfectly well without this header. This problem is because of the proxy you're using is relying on this header to mean something more than what it actually does. But sure, we could limit the use of the header to NTLM only to limit the scope of the use.

But again, for documentation and curiosity, do you know which proxy/version this is that causes us this problem?

[fajan]

Okay, I missunderstood.
Squid Cache: Version 2.7.5 as the header said.
Meanwhile I found that it also accepts the --proxy-header Connection:keep-alive, which should solve the other issue's as well. "Go's included ReverseProxy removes hop-by-hop HTTP headers, but forgot to remove the ancient and never-standardized "Proxy-Connection" hop-by-hop header."
I don't really know how other proxies handle to this.

 curl --proxy-ntlm --proxy proxy.estest.intra:3128 --proxy-header Connection:keep-alive --proxy-user MyDomain/ntlmtest:test -v https://example.com/file
*   Trying 101.102.103.104...
* Connected to proxy.estest.intra (101.102.103.104) port 3128 (#0)
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain/ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNT...AAAAAA=
> User-Agent: curl/7.50.1
> Connection:keep-alive
>
< HTTP/1.0 407 Proxy Authentication Required
< Server: squid/2.7.STABLE5
< Date: Fri, 12 Aug 2016 09:27:51 GMT
< Content-Type: text/html
< Content-Length: 1330
< X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
< Proxy-Authenticate: NTLM TlRMTVNTU......gAdQAAAAAA
< X-Cache: MISS from intra.my.domain.com
< X-Cache-Lookup: NONE from intra.my.domain.com:3128
< Via: 1.0 intra.my.domain.com:3128 (squid/2.7.STABLE5)
< Connection: keep-alive
< Proxy-Connection: keep-alive
<
* Ignore 1330 bytes of response-body
* TUNNEL_STATE switched to: 0
* Establish HTTP proxy tunnel to example.com:443
* Proxy auth using NTLM with user 'MyDomain/ntlmtest'
> CONNECT example.com:443 HTTP/1.1
> Host: example.com:443
> Proxy-Authorization: NTLM TlRMTVNTUAA......ISS1GRUpMMTM=
> User-Agent: curl/7.50.1
> Connection:keep-alive
>
< HTTP/1.1 200 Connection established
<
* Proxy replied OK to CONNECT request

[bagder]

Thanks for this, should be back and functional in git master now!