Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

curl can't connect thought NTLM proxy with --proxy-any option #876

Closed
NTMan opened this issue Jun 14, 2016 · 28 comments
Closed

curl can't connect thought NTLM proxy with --proxy-any option #876

NTMan opened this issue Jun 14, 2016 · 28 comments
Labels
authentication connecting & proxies HTTP KNOWN_BUGS material

Comments

@NTMan
Copy link

NTMan commented Jun 14, 2016

curl can't connect thought NTLM proxy with --proxy-any option

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl --version
curl 7.43.0 (x86_64-redhat-linux-gnu) libcurl/7.50.0-DEV NSS/3.24 Basic ECC zlib/1.2.8 libidn/1.32 libssh2/1.6.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets Metalink 

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl -v --proxy-any http://google.com
* Rebuilt URL to: http://google.com/
*   Trying 172.18.4.7...
* Connected to 172.18.4.7 (172.18.4.7) port 8080 (#0)
> GET http://google.com/ HTTP/1.1
> Host: google.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied.  )
< Via:1.1 ISAEG
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: Kerberos
< Proxy-Authenticate: Negotiate
< Connection: close
* HTTP/1.1 proxy connection set close!
< Proxy-Connection: close
< Pragma: no-cache
< Cache-Control: no-cache
< Content-Type: text/html
< Content-Length: 2374  
< 
* Closing connection 0
* Issue another request to this URL: 'http://google.com/'
* Hostname 172.18.4.7 was found in DNS cache
*   Trying 172.18.4.7...
* Connected to 172.18.4.7 (172.18.4.7) port 8080 (#1)
> GET http://google.com/ HTTP/1.1
> Host: google.com
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied.  )
< Via:1.1 ISAEG
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: Kerberos
* gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate. 
< Proxy-Authenticate: Negotiate
< Connection: close
* HTTP/1.1 proxy connection set close!
< Proxy-Connection: close
< Pragma: no-cache
< Cache-Control: no-cache
< Content-Type: text/html
< Content-Length: 2374  
< 
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML dir=ltr><HEAD><TITLE>The page cannot be displayed</TITLE>
<STYLE>A:link {
    FONT: 8pt/11pt verdana; COLOR: #ff0000
}
A:visited {
    FONT: 8pt/11pt verdana; COLOR: #4e4e4e
}
</STYLE>

<META content=NOINDEX name=ROBOTS>
<META http-equiv=Content-Type content="text-html; charset=Windows-1252">

<META content="MSHTML 5.50.4522.1800" name=GENERATOR></HEAD>
<BODY bgColor=#ffffff>
<TABLE cellSpacing=5 cellPadding=3 width=410>
  <TBODY>
  <TR>
    <TD vAlign=center align=left width=360>
      <H1 style="FONT: 13pt/15pt verdana; COLOR: #000000"><!--Problem-->The page 
      cannot be displayed</H1></TD></TR>
  <TR>
    <TD width=400 colSpan=2><FONT 
      style="FONT: 8pt/11pt verdana; COLOR: #000000">There is a problem with the 
      page you are trying to reach and it cannot be displayed.</FONT></TD></TR>
  <TR>
    <TD width=400 colSpan=2><FONT 
      style="FONT: 8pt/11pt verdana; COLOR: #000000">
      <HR color=#c0c0c0 noShade>

      <P>Please try the following:</P>
      <UL>
        <LI>Click the Refresh button, 
        or try again later.<BR>
        <LI>Open the Web site
         home page, and then look for links to the information you want. 
        <LI>If you typed the page address in the Address bar, make sure that it
        is spelled correctly.<BR>
        <LI>Verify that the Internet access policy on your network allows you
        to view this this page.</LI>
        <LI>If you believe you should be able to view this directory or page, 
        please contact the Web site administrator by using the e-mail address or 
        phone number listed on the Web site
         home page. </LI></UL>
      <H2 style="FONT: 8pt/11pt verdana; COLOR: #000000">HTTP 407 Proxy Authentication Required - The ISA Server requires authorization to fulfill the request. Access to the Web Proxy service is denied. (12209)<BR>Internet Security and Acceleration Server</H2>
      <HR color=#c0c0c0 noShade>

      <P>Technical Information (for support personnel)</P>
      <UL>
        <LI>Background:<BR>The gateway could not retrieve the requested page.<P></P></LI>
        <LI>ISA Server: isaeg.east.kronospan.int<BR>
        Via: <BR><BR>Time: 14.06.2016 14:09:49 GMT
        </LI></UL></FONT></TD></TR></TBODY></TABLE></BODY></HTML>
* Closing connection 1

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs /home/synergy/curl/curl/src/curl -V
curl 7.50.0-DEV (x86_64-unknown-linux-gnu) libcurl/7.50.0-DEV NSS/3.24 Basic ECC zlib/1.2.8 libidn/1.32 libssh2/1.6.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets Metalink �

Operating system Fedora 23 and RHEL 7.
This problem occurred if curl compiled with --with-gssapi option.
@iboukris
Copy link
Contributor

iboukris commented Jun 14, 2016 •
edited

  • gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate.

The proxy supports Negotiate which is prefered over NTLM so curl tries using GSSAPI and it fails.
You may use '--proxy-ntlm --proxy-basic' instead of any, to support both NTLM and Basic auth.

@NTMan
Copy link
Author

NTMan commented Jun 14, 2016 •
edited

Ok, why needed this mode. Some companies have corporate politics use Microsoft Forefront TMG.

And when we begin install Linux the problems begin. Many programs use the curl, but to tell them that they must use --proxy-ntlm option is impossible.
Maximum that I can tell to them is proxy address, user name and password. I like do it system-wide.

# cat /etc/profile.d/proxy.sh 
http_proxy=http://east-kronospan\\edvglu:niva4x4\$\$\$@172.18.4.7:8080
https_proxy=$http_proxy
export http_proxy
export https_proxy
no_proxy=127.0.0.1,10.18.1.57
export no_proxy

What we want?

We want that Linux workstation can work in Windows network. Able update via package manager (yum, dnf, aptitude, etc) download files via wget, send bug reports via gnome-abrt, and of course expected if it was a server all server software php, python etc able work via curl (SOAP) with external services of course without rewriting this software.

Here needed yet another option in environment which would be ignore GSSAPI faiure or ignore GSSAPI faiure by default.

Thanks.

@kdudka
Copy link
Contributor

kdudka commented Jun 14, 2016

On Tuesday, June 14, 2016 10:19:05 Mikhail wrote:

Ok, why needed this mode. Some companies have corporate politics use
Microsoft Forefront TMG.

And when we begin install Linux the problems begin. Many programs use the
curl, but to tell them that they must use --proxy-ntlm option is
impossible. Maximum that I can tell to them is proxy address, user name and
password. I like do it system-wide.

cat /etc/profile.d/proxy.sh

http_proxy=http://east-kronospan\edvglu:niva4x4\$\$\$@172.18.4.7:8080
https_proxy=$http_proxy
export http_proxy
export https_proxy
no_proxy=127.0.0.1,10.18.1.57
export no_proxy

What we want?

We want that Linux workstation can work in Windows network. Able update via
package manager (yum, dnf, aptitude, etc) download files via wget, send bug
reports via gnome-abrt, and of course expected if it was a server all
server software php, python atc able work via curl (SOAP) with external
services of course without rewriting this software.

Thank you for providing the background info. Could this be resolved
by configuring the proxy such that it does not advertise Kerberos?

Kamil

Here needed yet another option in environment which would be ignore GSSAPI
faiure or ignore GSSAPI faiure by default.

Thanks.

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#876 (comment)

@NTMan
Copy link
Author

NTMan commented Jun 15, 2016 •
edited

Unfortunately we can not affect admins which tuning the proxy server. If we could we already agreed about opening required ports to the Internet. We have already found a workaround this is cntlm proxy which installed on our side and pass all queries to NTLM proxy. But I would like that curl can do it directly. Linux should to be more friendlier and more comfortable this only way breaks through the way to desktop.

What is wrong with cntlm?

Main problem is installation this package, because package managers couldn't work (they use of course curl, but curl by default not work with NTLM), also not work wget. Only one way to install cntlm is manually download it with command curl --proxy-ntlm -O https://kojipkgs.fedoraproject.org/packages/cntlm/0.92.3/9.fc24/x86_64/cntlm-0.92.3-9.fc24.x86_64.rpm

Second problem for downloading cntlm package we must know full url to it.

For compare with Windows, for starting work enough enter proxy address and credentials.

@bagder bagder added the HTTP label Jun 20, 2016
@iboukris iboukris mentioned this issue Jun 22, 2016
Closed
@bagder
Copy link
Member

bagder commented Jun 23, 2016

gss_init_sec_context() failed: SPNEGO cannot find mechanisms to negotiate

That should of course be enough clues for libcurl to decide that it should rather try another allowed authentication mechanism...

@iboukris
Copy link
Contributor

Please see a possible fix for HTTP (works for me):
iboukris@148b911

It is a bit ugly because we clear 'data->state.authhost.avail' once we pick one auth from all the available ones (supported by the server), however we still have this info in 'data->info.httpauthavail'.

@iboukris
Copy link
Contributor

Hmm, I think this patch assumes Negotiate header is the first header.
Perhaps we need another approach.

@fajan fajan mentioned this issue Aug 11, 2016
Closed
@captain-caveman2k
Copy link
Contributor

captain-caveman2k commented Aug 21, 2016 •
edited

I've pushed a fix that I think addresses #718 for the most part which was built off some work I was doing back in March that I have recently pushed - However in this fix I have assumed we should let GSS-API based empty user names through as the user will be coming from the credientals cache.

I understand that even then it could fail so we could modify either this function or the new Curl_auth_is_spnego_supported() function to try and attempt to generate a token.

@captain-caveman2k
Copy link
Contributor

we clear 'data->state.authhost.avail' once we pick one auth from all the available ones

Does anyone know why we clear this here? It's been baffling me for a few days now!

@bagder
Copy link
Member

bagder commented Aug 21, 2016

Does anyone know why we clear this here

I don't recall exactly, but I think it may just be a precaution for the next request using the same easy handle?

@iboukris
Copy link
Contributor

I understand that even then it could fail so we could modify either this function or the new Curl_auth_is_spnego_supported() function to try and attempt to generate a token.

Yea, maybe we can call gss_inquire_cred() in the new Curl_auth_is_spnego_supported() check.

@iboukris
Copy link
Contributor

Actually, maybe it's better to attempt to generate a token instead of calling gss_inquire_cred() because it would help to detect possible failure due to improper target name, e.g. when IP is used instead of hostname (and no reverse dns available).

@NTMan
Copy link
Author

NTMan commented Jun 21, 2017

Any updates here? For me problem still actual for use Linux in Windows environment (AD + NTLM proxy).

@paulharris
Copy link

NTLM works for me if SPEGNO isn't compiled in, and I don't use the any-proxy option.
There are a few open issues on this topic.

@stale
Copy link

stale bot commented Dec 19, 2017

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Dec 19, 2017
@NTMan
Copy link
Author

NTMan commented Dec 19, 2017

Please not close this without fixing.

@stale stale bot removed the stale label Dec 19, 2017
@bagder
Copy link
Member

bagder commented Apr 30, 2018

Unless someone steps up and work on this issue, it will be added to KNOWN_BUGS and closed within soon.

@stale
Copy link

stale bot commented Oct 27, 2018

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Oct 27, 2018
@NTMan
Copy link
Author

NTMan commented Oct 27, 2018

Please not close this without fixing.

@stale stale bot removed the stale label Oct 27, 2018
@bagder
Copy link
Member

bagder commented Oct 27, 2018 •
edited

We will most likely close this and add a note to KNOWN_BUGS very soon, unless there's some progress. As explained.

@danielgustafsson
Copy link
Member

Please not close this without fixing.

If you can provide a reproducible test for this then the likelyhood that a developer will look at it increases dramatically.

@paulharris
Copy link

paulharris commented Oct 28, 2018 via email

@bagder
Copy link
Member

bagder commented Nov 7, 2018 •
edited

Without a reproducible test case we can't move anywhere on this issue and keeping it open by commenting every 6 months just annoys us all. I would strongly urge everyone who experiences this bug to work on setting up an easily reproducible set in order to get someone else to help out with debugging.

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl -v --proxy-any http://google.com

This is not a complete command line. --proxy-any asks for NTLM with the proxy, but you must also specify the proxy to use etc.

@NTMan
Copy link
Author

NTMan commented Nov 7, 2018

Without a reproducible test case we can't move anywhere on this issue and keeping it open by commenting every 6 months just annoys us all. I would strongly urge everyone who experiences this bug to work on setting up an easily reproducible set in order to get someone else to help out with debugging.

It not easy because such proxies only inhabit in Windows domain. So it also means that is impossible connect outside to such proxies.
Usually as Linux integrator we have a problem when we install our system in an organization where admins have a principled standpoint not to let anything pass to internet directly.
And we have no choice but to deal with Microsoft Forefront, which only auth by NTLM.

Therefore, I don’t even know how to help you. I think I will try to find somewhere Microsoft Forefront distribution to deploy it in our network.

$ LD_LIBRARY_PATH=/home/synergy/curl/curl/lib/.libs curl -v --proxy-any http://google.com

This is not a complete command line. --proxy-any asks for NTLM with the proxy, but you must also specify the proxy to use etc.

All proxy settings are configured via environment variables.

http_proxy=http://east-kronospan\\edvglu:niva4x4\$\$\$@172.18.4.7:8080
https_proxy=$http_proxy
export http_proxy
export https_proxy
no_proxy=127.0.0.1,10.18.1.57
export no_proxy

@paulharris
Copy link

paulharris commented Nov 7, 2018 via email

@danielgustafsson
Copy link
Member

danielgustafsson commented Nov 7, 2018 via email

@paulharris
Copy link

paulharris commented Nov 8, 2018 via email

@danielgustafsson
Copy link
Member

danielgustafsson commented Nov 8, 2018 •
edited

My recommendation here is that we close this issue as a known bug since we clearly aren't making any progress. The next step is to await someone sending an email to curl-library@ soliciting for access to an environment where this can be reproduced. If noone else sends it I suggest that either of you send it @paulharris @NTMan, being stakeholders in this. All we can do then is to hope that someone steps up and helps out.

I'll go ahead and open a PR for the KNOWN_BUGS file which when merged will close this issue, thereby connecting this discussion with the issue via the commit message.

danielgustafsson added a commit to danielgustafsson/curl that referenced this issue Nov 8, 2018
@danielgustafsson
KNOWN_BUGS: add --proxy-any connection issue
7beb209 
Add the identified issue with --proxy-any and proxy servers which
advertise authentication schemes other than the supported one.

Closes curl#876
Reported-by: NTMan on Github
@danielgustafsson danielgustafsson mentioned this issue Nov 8, 2018
Closed
@lock lock bot locked as resolved and limited conversation to collaborators Feb 7, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
authentication connecting & proxies HTTP KNOWN_BUGS material
Milestone
No milestone
Development

No branches or pull requests
7 participants
@bagder @NTMan @kdudka @danielgustafsson @captain-caveman2k @paulharris @iboukris