Git 2.6.2 and higher sending NTLM token instead of Kerberos when Negotiate is used

[damnhandy]

I am running into an issue where I am seeing Git 2.6.2 and higher on both Windows and Linux, where the Git client appears to be selecting NTLM instead of Kerberos. I am posting here as I have been unsuccessful at getting this issue posted to the Git mailing list. Kerberos support is Git on Windows is a primary concern. Curiously, GUI programs that are using the embedded Git 1.9.5.msysgit don't seem to have this issue and work correctly.

We are in the process of setting up a Git repository manager that is sitting behind an Nginx or Apache reverse proxy, which authenticates clients using Kerberos. From a general authentication perspective, kerberos appears to be working just fine as browsers and cURL are authenticated just fine. Some of our developers are using Atlasssian SourceTree (which uses an embedded version of git), and Kerberos authentication is working for them. Using Git 2.6.2 on the command line on the same system simply does not work.

On a Windows 7 system, I have set the GIT_CURL_VERBOSE=1 to see what is going on. Atlasssian SourceTree uses embedded Git 1.9.5.msysgit and when I issue a pull request to private repo, I get the following (token abbreviated):

* Adding handle: conn: 0x2fc2ac8
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 2 (0x2fc2ac8) send_pipe: 1, recv_pipe: 0
* Connected to myrepo.com (0.0.0.0) port 80 (#2)
* Server auth using GSS-Negotiate with user 'myuserid'
> GET /scm/repo/my-project.git/info/refs?service=git-upload-pack HTTP/1.1

Authorization: Negotiate YIILjgYGKwYBBQUCoIILgj (Removed for Brevity) 
    BoMAw==
User-Agent: git/1.9.5.msysgit.0
Host: myrepo.com
Accept: */*

Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Date: Fri, 30 Oct 2015 11:20:24 GMT
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< WWW-Authenticate: Negotiate jdhslkajhfljhasdlkjfhakljsdhfkljashdflkjahsjk
    lfhakljsdhflkah+BhDCBgaADAgEFoQMCAQ+ccccccccccccccccccccccccccccccccccc
    cccc/Iu/n/IGu7Jo8Y9xWY6Qa1sRidU6DkVUQIVYD0+rBRorrsxjkBd1N7mDlVltgg+jMzD
    xk9NY/JWhwIJqXPA/oI7yjlzJ8enqPG9gyHlqSreg==
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 UTC
< X-AREQUESTID: @9P77Cx680x198x0
< X-ASEN: SEN-L6706246
< X-AUSERID: 1
< X-AUSERNAME: myuserid
< X-ASESSIONID: 14716nd
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Expires: Tue, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Content-Type: application/x-git-upload-pack-advertisement
< Set-Cookie: JSESSIONID=55466CA87163DE4D5323977D7D64424C; Path=/; HttpOnly
< Via: 1.1 myrepo.com
< Connection: close
< Transfer-Encoding: chunked

With git/1.9.5.msysgit.0, everything works great, no issues.

On the same system using Git 2.6.2, I get the following:

PS C:\Users\myuserid> git clone http://myuserid@myrepo.com/scm/repo/
    my-repo.git 
Cloning into 'random-text-files'...
* Couldn't find host myrepo.com in the _netrc file; using defaults
* timeout on name lookup is not supported
*   Trying 0.0.0.0...
* Connected to myrepo.com (0.0.0.0) port 80 (#0)
> GET /scm/repo/my-repo.git/info/refs?service=git-upload-pack HTTP/1.1
Host: myrepo.com
User-Agent: git/2.6.2.windows.1
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Authorization Required
< Date: Fri, 30 Oct 2015 18:05:10 GMT
< WWW-Authenticate: Negotiate
< Content-Length: 503
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
* Closing connection 0
* Issue another request to this URL: 'http://myuserid@myrepo.com/scm/repo/
    my-repo.git/info/refs?service=git-upload-pack'
* Couldn't find host myrepo.com in the _netrc file; using defaults
* NTLM-proxy picked AND auth done set, clear picked!
* timeout on name lookup is not supported
* Hostname myrepo.com was found in DNS cache
*   Trying 0.0.0.0...
* Connected to myrepo.com (0.0.0.0) port 80 (#1)
> GET /scm/repo/my-repo.git/info/refs?service=git-upload-pack HTTP/1.1
Host: myrepo.com
User-Agent: git/2.6.2.windows.1
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Authorization Required
< Date: Fri, 30 Oct 2015 18:05:10 GMT
< WWW-Authenticate: Negotiate
< Content-Length: 503
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
* Closing connection 1
* Issue another request to this URL: 'http://myuserid@myrepo.com/scm/repo/
    my-repo.git/info/refs?service=git-upload-pack'
* Couldn't find host myrepo.com in the _netrc file; using defaults
* NTLM-proxy picked AND auth done set, clear picked!
* timeout on name lookup is not supported
* Hostname myrepo.com was found in DNS cache
*   Trying 0.0.0.0...
* Connected to myrepo.com (0.0.0.0) port 80 (#2)
* Server auth using Negotiate with user 'myuserid'
> GET /scm/repo/my-repo.git/info/refs?service=git-upload-pack HTTP/1.1
Host: myrepo.com
Authorization: Negotiate 
    TlRMTVNTUAABAAAAt4II4gXXXXXXXXXXXXXXXXXXXXXGAbEdAAAADw==
User-Agent: git/2.6.2.windows.1
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Authorization Required
< Date: Fri, 30 Oct 2015 18:05:10 GMT
< Content-Length: 503
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
* Closing connection 2
Password for 'http://myuserid@myrepo.com/scm/repo/my-repo.git':
* Couldn't find host myrepo.com in the _netrc file; using defaults
* NTLM-proxy picked AND auth done set, clear picked!
* timeout on name lookup is not supported
* Hostname myrepo.com was found in DNS cache
*   Trying 0.0.0.0...
* Connected to myrepo.com (0.0.0.0) port 80 (#3)
> GET /scm/repo/my-repo.git/info/refs?service=git-upload-pack HTTP/1.1
Host: myrepo.com
User-Agent: git/2.6.2.windows.1
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache

< HTTP/1.1 401 Authorization Required
< Date: Fri, 30 Oct 2015 18:05:15 GMT
< WWW-Authenticate: Negotiate
< Content-Length: 503
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
< 
* Closing connection 3
fatal: Authentication failed for 'http://myuserid@myrepo.com/scm/repo/
    my-repo.git/'

And here it fails. The authentication fails at the web server and it’s never hitting the Bitbucket Server behind it. I have tried this with Nginx and the spnego-http-auth-nginx-module. To rule out if it was something with the spnego-http-auth-nginx-module implementation, I have also tried it with Apache 2.2 and mod_auth_kerb and got similar results. Here is the server side logs from Apache:

[Fri Oct 30 13:14:14 2015] [debug] src/mod_auth_kerb.c(1279): 
    [client 10.23.6.40] Acquiring creds for HTTP/myrepo.com@MY.DOMAIN
[Fri Oct 30 13:14:14 2015] [debug] src/mod_auth_kerb.c(1692): 
    [client 10.23.6.40] Verifying client data using KRB5 GSS-API
[Fri Oct 30 13:14:14 2015] [debug] src/mod_auth_kerb.c(1708):
    [client 10.23.6.40] Client didn't delegate us their credential
[Fri Oct 30 13:14:14 2015] [debug] src/mod_auth_kerb.c(1736): 
    [client 10.23.6.40] Warning: received token seems to be NTLM, which 
    isn't supported by the Kerberos module. Check your IE configuration.

It would appear that the Git client is somehow defaulting to NTLM rather than Kerberos and causing things to break. The story is the same on the Linux side as well. Is there a similar environment variable in Git like GIT_CURL_VERBOSE that can used to control the authentication mechanism being used? Or is there some more information on how Git/libcurl make the determination to use NTLM vs Kerberos?

Ryan-

[rimrul]

I am running into an issue where I am seeing Git 2.6.2 and higher on both Windows and Linux, where the Git client appears to be selecting NTLM instead of Kerberos. I am posting here as I have been unsuccessful at getting this issue posted to the Git mailing list.

If you have posted this upstream and the issue exists on linux too, I don't think we will solve this.

Curiously, GUI programs that are using the embedded Git 1.9.5.msysgit don't seem to have this issue and work correctly.

You said the issue exists on 2.6.2 and newer. 1.9.5 is definitly lower than 2.6.2

[PhilipOakley]

Hi Ryan,
"unsuccessful at getting this issue posted to the Git mailing list " - This is usually because your email contains some HTML. There are also other magic words that it considers SPAM indicators as well. Even once you have got past that, the first posts may have some delay. But usually its your MUA/ISP sneaking in HTML!

If I remember correctly the list did have some discussion about similar authentication issues, with brian carlson being a contributor.

The mailing list archives are at http://news.gmane.org/gmane.comp.version-control.git with alternates at http://marc.info/?l=git&r=1&b=201403&w=2 or http://git.661346.n2.nabble.com/, so you should be able to find something via one of them.

Philip
----- Original Message -----
From: Ryan J. McDonough
To: git-for-windows/git
Sent: Saturday, January 16, 2016 3:44 PM
Subject: [git] Git 2.6.2 and higher sending NTLM token instead of Kerberos when Negotiate is used (#611)

I am running into an issue where I am seeing Git 262 on both Windows, and Linux, where the Git client appears to be selecting NTLM instead of Kerberos I am posting here as I have been unsuccessful at getting this issue posted to the Git mailing list Kerberos support is Git on Windows is a primary concern Curiously, GUI programs that are using the embedded Git 195msysgit don't seem to have this issue and work correctly

We are in the process of setting up a Git repository manager that is sitting behind an Nginx or Apache reverse proxy, which authenticates clients using Kerberos From a general authentication perspective, kerberos appears to be working just fine as browsers and cURL are authenticated just fine Some of our developers are using Atlasssian SourceTree (which uses an embedded version of git), and Kerberos authentication is working for them Using Git 262 on the command line on the same system simply does not work

On a Windows 7 system, I have set the GIT_CURL_VERBOSE=1 to see what is going on Atlasssian SourceTree uses embedded Git 195msysgit and when I issue a pull request to private repo, I get the following (token abbreviated):

• Adding handle: conn: 0x2fc2ac8
• Adding handle: send: 0
• Adding handle: recv: 0
• Curl_addHandleToPipeline: length: 1
• - Conn 2 (0x2fc2ac8) send_pipe: 1, recv_pipe: 0
• Connected to myrepocom (0000) port 80 (t9300: use test_cmp_bin instead of test_cmp to compare binary files #2)
• Server auth using GSS-Negotiate with user 'myuserid'

  GET /scm/repo/my-projectgit/info/refs?service=git-upload-pack HTTP/11

Authorization: Negotiate YIILjgYGKwYBBQUCoIILgj (Removed for Brevity)
BoMAw==
User-Agent: git/195msysgit0
Host: myrepocom
Accept: /

Accept-Encoding: gzip
Pragma: no-cache

< HTTP/11 200 OK
< Date: Fri, 30 Oct 2015 11:20:24 GMT

• Server Apache-Coyote/11 is not blacklisted
  < Server: Apache-Coyote/11
  < WWW-Authenticate: Negotiate jdhslkajhfljhasdlkjfhakljsdhfkljashdflkjahsjk
  lfhakljsdhflkah+BhDCBgaADAgEFoQMCAQ+ccccccccccccccccccccccccccccccccccc
  cccc/Iu/n/IGu7Jo8Y9xWY6Qa1sRidU6DkVUQIVYD0+rBRorrsxjkBd1N7mDlVltgg+jMzD
  xk9NY/JWhwIJqXPA/oI7yjlzJ8enqPG9gyHlqSreg==
  < Cache-Control: private
  < Expires: Thu, 01 Jan 1970 00:00:00 UTC
  < X-AREQUESTID: @9P77Cx680x198x0
  < X-ASEN: SEN-L6706246
  < X-AUSERID: 1
  < X-AUSERNAME: myuserid
  < X-ASESSIONID: 14716nd
  < X-XSS-Protection: 1; mode=block
  < X-Frame-Options: SAMEORIGIN
  < X-Content-Type-Options: nosniff
  < Expires: Tue, 01 Jan 1980 00:00:00 GMT
  < Pragma: no-cache
  < Cache-Control: no-cache, max-age=0, must-revalidate
  < Content-Type: application/x-git-upload-pack-advertisement
  < Set-Cookie: JSESSIONID=55466CA87163DE4D5323977D7D64424C; Path=/; HttpOnly
  < Via: 11 myrepocom
  < Connection: close
  < Transfer-Encoding: chunked
  With git/195msysgit0, everything works great, no issues

  On the same system using Git 262, I get the following:

PS C:\Users\myuserid> git clone http://myuserid@myrepocom/scm/repo/
my-repogit
Cloning into 'random-text-files'

• Couldn't find host myrepocom in the _netrc file; using defaults
• timeout on name lookup is not supported
• Trying 0000
• Connected to myrepocom (0000) port 80 (#0)

  GET /scm/repo/my-repogit/info/refs?service=git-upload-pack HTTP/11
  Host: myrepocom
  User-Agent: git/262windows1
  Accept: /
  Accept-Encoding: gzip
  Accept-Language: en-US, *;q=09
  Pragma: no-cache

< HTTP/11 401 Authorization Required
< Date: Fri, 30 Oct 2015 18:05:10 GMT
< WWW-Authenticate: Negotiate
< Content-Length: 503
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<

• Closing connection 0
• Issue another request to this URL: 'http://myuserid@myrepocom/scm/repo/
  my-repogit/info/refs?service=git-upload-pack'
• Couldn't find host myrepocom in the _netrc file; using defaults
• NTLM-proxy picked AND auth done set, clear picked!
• timeout on name lookup is not supported
• Hostname myrepocom was found in DNS cache
• Trying 0000
• Connected to myrepocom (0000) port 80 (Fixes for the new SDK #1)

  GET /scm/repo/my-repogit/info/refs?service=git-upload-pack HTTP/11
  Host: myrepocom
  User-Agent: git/262windows1
  Accept: /
  Accept-Encoding: gzip
  Accept-Language: en-US, *;q=09
  Pragma: no-cache

< HTTP/11 401 Authorization Required
< Date: Fri, 30 Oct 2015 18:05:10 GMT
< WWW-Authenticate: Negotiate
< Content-Length: 503
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<

• Closing connection 1
• Issue another request to this URL: 'http://myuserid@myrepocom/scm/repo/
  my-repogit/info/refs?service=git-upload-pack'
• Couldn't find host myrepocom in the _netrc file; using defaults
• NTLM-proxy picked AND auth done set, clear picked!
• timeout on name lookup is not supported
• Hostname myrepocom was found in DNS cache
• Trying 0000
• Connected to myrepocom (0000) port 80 (t9300: use test_cmp_bin instead of test_cmp to compare binary files #2)
• Server auth using Negotiate with user 'myuserid'

  GET /scm/repo/my-repogit/info/refs?service=git-upload-pack HTTP/11
  Host: myrepocom
  Authorization: Negotiate
  TlRMTVNTUAABAAAAt4II4gXXXXXXXXXXXXXXXXXXXXXGAbEdAAAADw==
  User-Agent: git/262windows1
  Accept: /
  Accept-Encoding: gzip
  Accept-Language: en-US, *;q=09
  Pragma: no-cache

< HTTP/11 401 Authorization Required
< Date: Fri, 30 Oct 2015 18:05:10 GMT
< Content-Length: 503
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<

• Closing connection 2
  Password for 'http://myuserid@myrepocom/scm/repo/my-repogit':
• Couldn't find host myrepocom in the _netrc file; using defaults
• NTLM-proxy picked AND auth done set, clear picked!
• timeout on name lookup is not supported
• Hostname myrepocom was found in DNS cache
• Trying 0000
• Connected to myrepocom (0000) port 80 (Final test fixes #3)

  GET /scm/repo/my-repogit/info/refs?service=git-upload-pack HTTP/11
  Host: myrepocom
  User-Agent: git/262windows1
  Accept: /
  Accept-Encoding: gzip
  Accept-Language: en-US, *;q=09
  Pragma: no-cache

< HTTP/11 401 Authorization Required
< Date: Fri, 30 Oct 2015 18:05:15 GMT
< WWW-Authenticate: Negotiate
< Content-Length: 503
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<

• Closing connection 3
  fatal: Authentication failed for 'http://myuserid@myrepocom/scm/repo/
  my-repogit/'
  And here it fails The authentication fails at the web server and it’s never hitting the Bitbucket Server behind it I have tried this with Nginx and the spnego-http-auth-nginx-module To rule out if it was something with the spnego-http-auth-nginx-module implementation, I have also tried it with Apache 22 and mod_auth_kerb and got similar results Here is the server side logs from Apache:

[Fri Oct 30 13:14:14 2015] [debug] src/mod_auth_kerbc(1279):
[client 1023640] Acquiring creds for HTTP/myrepocom@MYDOMAIN
[Fri Oct 30 13:14:14 2015] [debug] src/mod_auth_kerbc(1692):
[client 1023640] Verifying client data using KRB5 GSS-API
[Fri Oct 30 13:14:14 2015] [debug] src/mod_auth_kerbc(1708):
[client 1023640] Client didn't delegate us their credential
[Fri Oct 30 13:14:14 2015] [debug] src/mod_auth_kerbc(1736):
[client 1023640] Warning: received token seems to be NTLM, which
isn't supported by the Kerberos module Check your IE configuration
It would appear that the Git client is somehow defaulting to NTLM rather than Kerberos and causing things to break The story is the same on the Linux side as well Is there a similar environment variable in Git like GIT_CURL_VERBOSE that can used to control the authentication mechanism being used? Or is there some more information on how Git/libcurl make the determination to use NTLM vs Kerberos?

Ryan-

—
Reply to this email directly or view it on GitHub.

[chris-araman]

Might be related to this?
curl/curl#520

[nightman68]

@damnhandy when I compare the communication between the 2 releases for me it looks like that the older release is not passing any proxy but the new is passing a proxy which supports only NTLM authentication.
I'm not a proxy expert but I think that the proxy is not forwarding the Kerberos token to the Web server, the NTLM token is forwarded.
Did you tried to disable the proxy? Do both releases are using the same configuration file?

[dscho]

@damnhandy have you tried to follow @PhilipOakley's advice?

[kkpattern]

We also encountered this problem with git/2.24.1.windows.2. Is there any way to fix this? We really want to use kerberos auth on Windows.