curl can't connect thought NTLM proxy with --proxy-any option #876

NTMan · Jun 14, 2016

curl can't connect thought NTLM proxy with --proxy-any option

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl --version
curl 7.43.0 (x86_64-redhat-linux-gnu) libcurl/7.50.0-DEV NSS/3.24 Basic ECC zlib/1.2.8 libidn/1.32 libssh2/1.6.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets Metalink 

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl -v --proxy-any http://google.com
* Rebuilt URL to: http://google.com/
*   Trying 172.18.4.7...
* Connected to 172.18.4.7 (172.18.4.7) port 8080 (#0)
> GET http://google.com/ HTTP/1.1
> Host: google.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied.  )
< Via:1.1 ISAEG
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: Kerberos
< Proxy-Authenticate: Negotiate
< Connection: close
* HTTP/1.1 proxy connection set close!
< Proxy-Connection: close
< Pragma: no-cache
< Cache-Control: no-cache
< Content-Type: text/html
< Content-Length: 2374  
< 
* Closing connection 0
* Issue another request to this URL: 'http://google.com/'
* Hostname 172.18.4.7 was found in DNS cache
*   Trying 172.18.4.7...
* Connected to 172.18.4.7 (172.18.4.7) port 8080 (#1)
> GET http://google.com/ HTTP/1.1
> Host: google.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied.  )
< Via:1.1 ISAEG
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: Kerberos
* gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate. 
< Proxy-Authenticate: Negotiate
< Connection: close
* HTTP/1.1 proxy connection set close!
< Proxy-Connection: close
< Pragma: no-cache
< Cache-Control: no-cache
< Content-Type: text/html
< Content-Length: 2374  
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML dir=ltr><HEAD><TITLE>The page cannot be displayed</TITLE>
<STYLE>A:link {
    FONT: 8pt/11pt verdana; COLOR: #ff0000
}
A:visited {
    FONT: 8pt/11pt verdana; COLOR: #4e4e4e
}
</STYLE>

<META content=NOINDEX name=ROBOTS>
<META http-equiv=Content-Type content="text-html; charset=Windows-1252">

<META content="MSHTML 5.50.4522.1800" name=GENERATOR></HEAD>
<BODY bgColor=#ffffff>
<TABLE cellSpacing=5 cellPadding=3 width=410>
  <TBODY>
  <TR>
    <TD vAlign=center align=left width=360>
      <H1 style="FONT: 13pt/15pt verdana; COLOR: #000000"><!--Problem-->The page 
      cannot be displayed</H1></TD></TR>
  <TR>
    <TD width=400 colSpan=2><FONT 
      style="FONT: 8pt/11pt verdana; COLOR: #000000">There is a problem with the 
      page you are trying to reach and it cannot be displayed.</FONT></TD></TR>
  <TR>
    <TD width=400 colSpan=2><FONT 
      style="FONT: 8pt/11pt verdana; COLOR: #000000">
      <HR color=#c0c0c0 noShade>

      <P>Please try the following:</P>
      <UL>
        <LI>Click the Refresh button, 
        or try again later.<BR>
        <LI>Open the Web site
         home page, and then look for links to the information you want. 
        <LI>If you typed the page address in the Address bar, make sure that it
        is spelled correctly.<BR>
        <LI>Verify that the Internet access policy on your network allows you
        to view this this page.</LI>
        <LI>If you believe you should be able to view this directory or page, 
        please contact the Web site administrator by using the e-mail address or 
        phone number listed on the Web site
         home page. </LI></UL>
      <H2 style="FONT: 8pt/11pt verdana; COLOR: #000000">HTTP 407 Proxy Authentication Required - The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. (12209)<BR>Internet Security and Acceleration Server</H2>
      <HR color=#c0c0c0 noShade>

      <P>Technical Information (for support personnel)</P>
      <UL>
        <LI>Background:<BR>The gateway could not retrieve the requested page.<P></P></LI>
        <LI>ISA Server: isaeg.east.kronospan.int<BR>
        Via: <BR><BR>Time: 14.06.2016 14:09:49 GMT
        </LI></UL></FONT></TD></TR></TBODY></TABLE></BODY></HTML>
* Closing connection 1

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs /home/synergy/curl/curl/src/curl -V
curl 7.50.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.50.0-DEV NSS/3.24 Basic ECC zlib/1.2.8 libidn/1.32 libssh2/1.6.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets Metalink �

Operating system Fedora 23 and RHEL 7.
This problem occurred if curl compiled with --with-gssapi option.

iboukris · Jun 14, 2016

gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate.

The proxy supports Negotiate which is prefered over NTLM so curl tries using GSSAPI and it fails.
You may use '--proxy-ntlm --proxy-basic' instead of any, to support both NTLM and Basic auth.

NTMan · Jun 14, 2016

Ok, why needed this mode. Some companies have corporate politics use Microsoft Forefront TMG.

And when we begin install Linux the problems begin. Many programs use the curl, but to tell them that they must use --proxy-ntlm option is impossible.
Maximum that I can tell to them is proxy address, user name and password. I like do it system-wide.

# cat /etc/profile.d/proxy.sh 
http_proxy=http://east-kronospan\\edvglu:niva4x4\$\$\$@172.18.4.7:8080
https_proxy=$http_proxy
export http_proxy
export https_proxy
no_proxy=127.0.0.1,10.18.1.57
export no_proxy

What we want?

We want that Linux workstation can work in Windows network. Able update via package manager (yum, dnf, aptitude, etc) download files via wget, send bug reports via gnome-abrt, and of course expected if it was a server all server software php, python etc able work via curl (SOAP) with external services of course without rewriting this software.

Here needed yet another option in environment which would be ignore GSSAPI faiure or ignore GSSAPI faiure by default.

Thanks.

kdudka · Jun 14, 2016

On Tuesday, June 14, 2016 10:19:05 Mikhail wrote:

Ok, why needed this mode. Some companies have corporate politics use
Microsoft Forefront TMG.

And when we begin install Linux the problems begin. Many programs use the
curl, but to tell them that they must use --proxy-ntlm option is
impossible. Maximum that I can tell to them is proxy address, user name and
password. I like do it system-wide.

cat /etc/profile.d/proxy.sh

http_proxy=http://east-kronospan\edvglu:niva4x4\$\$\$@172.18.4.7:8080
https_proxy=$http_proxy
export http_proxy
export https_proxy
no_proxy=127.0.0.1,10.18.1.57
export no_proxy

What we want?

We want that Linux workstation can work in Windows network. Able update via
package manager (yum, dnf, aptitude, etc) download files via wget, send bug
reports via gnome-abrt, and of course expected if it was a server all
server software php, python atc able work via curl (SOAP) with external
services of course without rewriting this software.

Thank you for providing the background info. Could this be resolved
by configuring the proxy such that it does not advertise Kerberos?

Kamil

Here needed yet another option in environment which would be ignore GSSAPI
faiure or ignore GSSAPI faiure by default.

Thanks.

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#876 (comment)

NTMan · Jun 15, 2016

Unfortunately we can not affect admins which tuning the proxy server. If we could we already agreed about opening required ports to the Internet. We have already found a workaround this is cntlm proxy which installed on our side and pass all queries to NTLM proxy. But I would like that curl can do it directly. Linux should to be more friendlier and more comfortable this only way breaks through the way to desktop.

What is wrong with cntlm?

Main problem is installation this package, because package managers couldn't work (they use of course curl, but curl by default not work with NTLM), also not work wget. Only one way to install cntlm is manually download it with command curl --proxy-ntlm -O https://kojipkgs.fedoraproject.org/packages/cntlm/0.92.3/9.fc24/x86_64/cntlm-0.92.3-9.fc24.x86_64.rpm

Second problem for downloading cntlm package we must know full url to it.

For compare with Windows, for starting work enough enter proxy address and credentials.

bagder · Jun 23, 2016

gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate

That should of course be enough clues for libcurl to decide that it should rather try another allowed authentication mechanism...

iboukris · Jun 23, 2016

Please see a possible fix for HTTP (works for me):
iboukris@148b911

It is a bit ugly because we clear 'data->state.authhost.avail' once we pick one auth from all the available ones (supported by the server), however we still have this info in 'data->info.httpauthavail'.

iboukris · Jun 25, 2016

Hmm, I think this patch assumes Negotiate header is the first header.
Perhaps we need another approach.

captain-caveman2k · Aug 21, 2016

I've pushed a fix that I think addresses #718 for the most part which was built off some work I was doing back in March that I have recently pushed - However in this fix I have assumed we should let GSS-API based empty user names through as the user will be coming from the credientals cache.

I understand that even then it could fail so we could modify either this function or the new Curl_auth_is_spnego_supported() function to try and attempt to generate a token.

captain-caveman2k · Aug 21, 2016

we clear 'data->state.authhost.avail' once we pick one auth from all the available ones

Does anyone know why we clear this here? It's been baffling me for a few days now!

bagder · Aug 21, 2016

Does anyone know why we clear this here

I don't recall exactly, but I think it may just be a precaution for the next request using the same easy handle?

iboukris · Aug 21, 2016

I understand that even then it could fail so we could modify either this function or the new Curl_auth_is_spnego_supported() function to try and attempt to generate a token.

Yea, maybe we can call gss_inquire_cred() in the new Curl_auth_is_spnego_supported() check.

iboukris · Aug 21, 2016

Actually, maybe it's better to attempt to generate a token instead of calling gss_inquire_cred() because it would help to detect possible failure due to improper target name, e.g. when IP is used instead of hostname (and no reverse dns available).

NTMan · Jun 21, 2017

Any updates here? For me problem still actual for use Linux in Windows environment (AD + NTLM proxy).

paulharris · Jun 22, 2017

NTLM works for me if SPEGNO isn't compiled in, and I don't use the any-proxy option.
There are a few open issues on this topic.

stale · Dec 19, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

NTMan · Dec 19, 2017

Please not close this without fixing.

bagder · Apr 30, 2018

Unless someone steps up and work on this issue, it will be added to KNOWN_BUGS and closed within soon.

stale · Oct 27, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

NTMan · Oct 27, 2018

Please not close this without fixing.

bagder · Oct 27, 2018

We will most likely close this and add a note to KNOWN_BUGS very soon, unless there's some progress. As explained.

danielgustafsson · Oct 27, 2018

Please not close this without fixing.

If you can provide a reproducible test for this then the likelyhood that a developer will look at it increases dramatically.

paulharris · Oct 28, 2018

That would require a test environment with an ntlm proxy, do any developers have one? Or can acquire one?

…

On Sun., 28 Oct. 2018, 2:30 am Daniel Gustafsson, ***@***.***> wrote: Please not close this without fixing. If you can provide a reproducible test for this then the likelyhood that a developer will look at it increases dramatically. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#876 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABkgS8qyFGjvRC2zxF6rR3JhOZ2t79Ajks5upKY3gaJpZM4I1dPY> .

bagder · Nov 7, 2018

Without a reproducible test case we can't move anywhere on this issue and keeping it open by commenting every 6 months just annoys us all. I would strongly urge everyone who experiences this bug to work on setting up an easily reproducible set in order to get someone else to help out with debugging.

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl -v --proxy-any http://google.com

This is not a complete command line. --proxy-any asks for NTLM with the proxy, but you must also specify the proxy to use etc.

NTMan · Nov 7, 2018

Without a reproducible test case we can't move anywhere on this issue and keeping it open by commenting every 6 months just annoys us all. I would strongly urge everyone who experiences this bug to work on setting up an easily reproducible set in order to get someone else to help out with debugging.

It not easy because such proxies only inhabit in Windows domain. So it also means that is impossible connect outside to such proxies.
Usually as Linux integrator we have a problem when we install our system in an organization where admins have a principled standpoint not to let anything pass to internet directly.
And we have no choice but to deal with Microsoft Forefront, which only auth by NTLM.

Therefore, I don’t even know how to help you. I think I will try to find somewhere Microsoft Forefront distribution to deploy it in our network.

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl -v --proxy-any http://google.com

This is not a complete command line. --proxy-any asks for NTLM with the proxy, but you must also specify the proxy to use etc.

All proxy settings are configured via environment variables.

http_proxy=http://east-kronospan\\edvglu:niva4x4\$\$\$@172.18.4.7:8080
https_proxy=$http_proxy
export http_proxy
export https_proxy
no_proxy=127.0.0.1,10.18.1.57
export no_proxy

paulharris · Nov 7, 2018

I've never set up an ntlm proxy, Only suffered a long way down stream from them. I did spin up a Microsoft cloud computer to test an sql server issue, Does anyone know if we could get forefront running on something like that? or whatever the proxy software is called. Perhaps if the curl website could put out a call to some friendly corporate supporter to help in this regard? Microsoft has Ubuntu running on Windows now, perhaps an opportunity there? I don't have the personal connections to help set up an environment you could use for a test case, But hopefully the curl software is so popular that there is some connection that way?

…

On Wed., 7 Nov. 2018, 4:56 pm Mikhail ***@***.*** wrote: Without a reproducible test case we can't move anywhere on this issue and keeping it open by commenting every 6 months just annoys us all. I would *strongly urge* everyone who experiences this bug to work on setting up an easily reproducible set in order to get someone else to help out with debugging. It not easy because such proxies only inhabit in Windows domain. So it also means that is impossible connect outside to such proxies. Usually as Linux integrator we have a problem when we install our system in an organization where admins have a principled standpoint not to let anything pass to internet directly. And we have no choice but to deal with Microsoft Forefront, which only auth by NTLM. Therefore, I don’t even know how to help you. I think I will try to find somewhere Microsoft Forefront distribution to deploy it in our network. $ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl -v --proxy-any http://google.com This is not a complete command line. --proxy-any asks for NTLM with the proxy, but you must also specify the proxy to use etc. All proxy settings are configured via environment variables. ***@***.***:8080 https_proxy=$http_proxy export http_proxy export https_proxy no_proxy=127.0.0.1,10.18.1.57 export no_proxy — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#876 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABkgS67kOdSespEWAZGxTApKLI4pNQp4ks5usqBLgaJpZM4I1dPY> .

danielgustafsson · Nov 7, 2018

On 7 Nov 2018, at 14:05, paulharris ***@***.***> wrote: Perhaps if the curl website could put out a call to some friendly corporate supporter to help in this regard?

I’m not sure how effective that would be to be honest, the individuals who has the ability to set something like this us might not be casually browsing the curl website. More effective is probably if those affected by this could raise such a call for participation on the curl-library mailing list, and then spread the word linking to the discussion there. The odds of “the right people” being subscribed to the mailinglist is probably higher.

paulharris · Nov 8, 2018

Regarding the email, it would be better if it came from the developers who will be running these tests. What do you guys need ? What will you work with? If I email the list, I'll be acting as middle man, because what I would like is probably not what you would like. I only hack on libcurl when I need to move past a bug, but this environment is required for the longer term. Also I want to point out that in a Windows environment, the user should not have to specify a proxy. Windows has a proxy detection system, and as a result, users likely don't know what the proxy is. Example of how to do it is here: https://github.com/paulharris/libdetectproxy On Wed, 7 Nov 2018 at 21:17, Daniel Gustafsson <notifications@github.com> wrote:

…

> On 7 Nov 2018, at 14:05, paulharris ***@***.***> wrote: > Perhaps if the curl website could put out a call to some friendly corporate > supporter to help in this regard? I’m not sure how effective that would be to be honest, the individuals who has the ability to set something like this us might not be casually browsing the curl website. More effective is probably if those affected by this could raise such a call for participation on the curl-library mailing list, and then spread the word linking to the discussion there. The odds of “the right people” being subscribed to the mailinglist is probably higher. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#876 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABkgS1ONkW9PGMxcd1XyCMNJouo0FvOXks5ust1ygaJpZM4I1dPY> .

danielgustafsson · Nov 8, 2018

My recommendation here is that we close this issue as a known bug since we clearly aren't making any progress. The next step is to await someone sending an email to curl-library@ soliciting for access to an environment where this can be reproduced. If noone else sends it I suggest that either of you send it @paulharris @NTMan, being stakeholders in this. All we can do then is to hope that someone steps up and helps out.

I'll go ahead and open a PR for the KNOWN_BUGS file which when merged will close this issue, thereby connecting this discussion with the issue via the commit message.

bagder added the HTTP label Jun 20, 2016

iboukris referenced this issue Jun 22, 2016
Closed
POP3 NTLM authentication got broken in between 7.37 and 7.38 #718

fajan referenced this issue Aug 11, 2016
Closed
proxy-ntlm requires Proxy-Connection: keep-alive #954

bagder added Authentication connecting & proxies labels Feb 13, 2017

stale bot added the stale label Dec 19, 2017

stale bot removed the stale label Dec 19, 2017

bagder added the KNOWN_BUGS material label Apr 30, 2018

tmds referenced this issue Jul 11, 2018
Closed
Http Negotiate authentication fails on Unix #31005

stale bot added the stale label Oct 27, 2018

stale bot removed the stale label Oct 27, 2018

danielgustafsson referenced this issue Nov 8, 2018
Closed
KNOWN_BUGS: add --proxy-any connection issue #3250

danielgustafsson closed this in 10c91b6 Nov 9, 2018

curl/curl

Join GitHub today

curl can't connect thought NTLM proxy with --proxy-any option #876

Comments

NTMan commented Jun 14, 2016

This comment has been minimized.

iboukris commented Jun 14, 2016 • edited

This comment has been minimized.

NTMan commented Jun 14, 2016 • edited

This comment has been minimized.

kdudka commented Jun 14, 2016

cat /etc/profile.d/proxy.sh

This comment has been minimized.

NTMan commented Jun 15, 2016 • edited

bagder added the HTTP label Jun 20, 2016

iboukris referenced this issue Jun 22, 2016

POP3 NTLM authentication got broken in between 7.37 and 7.38 #718

This comment has been minimized.

bagder commented Jun 23, 2016

This comment has been minimized.

iboukris commented Jun 23, 2016

This comment has been minimized.

iboukris commented Jun 25, 2016

fajan referenced this issue Aug 11, 2016

proxy-ntlm requires Proxy-Connection: keep-alive #954

This comment has been minimized.

captain-caveman2k commented Aug 21, 2016 • edited

This comment has been minimized.

captain-caveman2k commented Aug 21, 2016

This comment has been minimized.

bagder commented Aug 21, 2016

This comment has been minimized.

iboukris commented Aug 21, 2016

This comment has been minimized.

iboukris commented Aug 21, 2016

bagder added Authentication connecting & proxies labels Feb 13, 2017

This comment has been minimized.

NTMan commented Jun 21, 2017

This comment has been minimized.

paulharris commented Jun 22, 2017

This comment has been minimized.

stale bot commented Dec 19, 2017

stale bot added the stale label Dec 19, 2017

This comment has been minimized.

NTMan commented Dec 19, 2017

stale bot removed the stale label Dec 19, 2017

bagder added the KNOWN_BUGS material label Apr 30, 2018

This comment has been minimized.

bagder commented Apr 30, 2018

tmds referenced this issue Jul 11, 2018

Http Negotiate authentication fails on Unix #31005

This comment has been minimized.

stale bot commented Oct 27, 2018

stale bot added the stale label Oct 27, 2018

This comment has been minimized.

NTMan commented Oct 27, 2018

stale bot removed the stale label Oct 27, 2018

This comment has been minimized.

bagder commented Oct 27, 2018 • edited

This comment has been minimized.

danielgustafsson commented Oct 27, 2018

This comment has been minimized.

paulharris commented Oct 28, 2018

This comment has been minimized.

bagder commented Nov 7, 2018 • edited

This comment has been minimized.

NTMan commented Nov 7, 2018

This comment has been minimized.

paulharris commented Nov 7, 2018

This comment has been minimized.

danielgustafsson commented Nov 7, 2018

This comment has been minimized.

paulharris commented Nov 8, 2018

This comment has been minimized.

danielgustafsson commented Nov 8, 2018 • edited

danielgustafsson added a commit to danielgustafsson/curl that referenced this issue Nov 8, 2018

danielgustafsson referenced this issue Nov 8, 2018

KNOWN_BUGS: add --proxy-any connection issue #3250

danielgustafsson closed this in 10c91b6 Nov 9, 2018

iboukris commented Jun 14, 2016 •

edited

NTMan commented Jun 14, 2016 •

edited

NTMan commented Jun 15, 2016 •

edited

captain-caveman2k commented Aug 21, 2016 •

edited

bagder commented Oct 27, 2018 •

edited

bagder commented Nov 7, 2018 •

edited

danielgustafsson commented Nov 8, 2018 •

edited

danielgustafsson closed this in `10c91b6` Nov 9, 2018