Reverse proxy on ASP.NET sites using NTLM authentication is not working #663

shytikov · 2016-03-08T12:22:48Z

1. What version of Caddy are you running (`caddy -version`)?

Caddy 0.8.2

2. What are you trying to do?

I'm trying to use Caddy as reverse proxy for MS Dynamics CRM installation to get more robust and balance loading, better performance (maybe) by applying HTTP/2 on context served by MS CRM.

3. What is your entire Caddyfile?

localhost:2020
gzip
log access.log
proxy / https://azure.based.installation.of/mscrm/

4. How did you run Caddy (give the full command and describe the execution environment)?

Just typed caddy in cmd.exe
I'm using Windows 8 / 64;
I'm tested scenario in whole zoo of browsers: FF, IE, Chrome most recent version available;
MS CRM hosted on Azure.
MS CRM requires authentication, and normally uses simple Windows NTML.
No IFD (Internet Facing Deployment) is used (it's more or less cookie-based auth mechanism);

5. What did you expect to see?

I expected Caddy to be completely transparent, like Fiddler2, for example, normally does.

6. What did you see instead (give full error messages and/or log)?

Instead I get never ending sequence of requests asking to enter my credentials.

Replying with 401 is normal behavior for NTLM auth, but as soon as browser supplies correct credentials, token is issued and these 401 re-directions stop. But not in this case. I suppose that Caddy misses to transfer some important header or any other information is not substituted correctly. As result, transparency of proxy literally becomes opaque.

I don't think that information in Caddy's access log is valuable here, since information about headers there is not present, as I can see — but I looked through it briefly, and URL used are 100% correct.

But. I can provide RAW output from fiddler for investigation. For privacy reasons I will replace actual urls and authorization tokens (and yes, I have treble checked my password, I'm entering it without spelling errors).

Request:

GET http://localhost:2020/[removed]/_common/styles/global.css.aspx?lcid=1053&ver=-2039462857 HTTP/1.1
Accept: text/css, */*
Referer: http://localhost:2020/
Accept-Language: en-US,en;q=0.7,ru;q=0.3
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko
Accept-Encoding: gzip, deflate
Connection: Keep-Alive
DNT: 1
Host: localhost:2020
Authorization: Negotiate [removed]

Response:

HTTP/1.1 401 Unauthorized
Content-Length: 341
Content-Type: text/html; charset=us-ascii
Date: Tue, 08 Mar 2016 11:57:39 GMT
Server: Caddy
Server: Microsoft-HTTPAPI/2.0
Www-Authenticate: Negotiate [removed]
Proxy-Support: Session-Based-Authentication

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Not Authorized</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Not Authorized</h2>
<hr><p>HTTP Error 401. The requested resource requires user authentication.</p>
</BODY></HTML>

The text was updated successfully, but these errors were encountered:

mholt · 2016-03-08T19:33:59Z

Thanks for the clear bug report!

I expected Caddy to be completely transparent, like Fiddler2, for example, normally does.

Having not used Fiddler2, do you meant that you expect Caddy to proxy all the headers upstream by default?

You can write header values out to the log using a log format with {>Header-Name}, for example. That might give you more information. There is the proxy_header rule you can add to the proxy directive to add those headers, if that is what is missing.

Unfortunately I'm not at all familiar with MS Dynamics CRM, but I bet we can figure this out. (Maybe somebody else has more experience with it too?)

shytikov · 2016-03-08T19:55:47Z

Thanks for reply!

I have added sample request / response pair I captured with fiiddler (you can find it in my post above). I will compare it with normal communication and will try to find out what's missing.

But for me it looks like NTML authentication gets broken somehow by Caddy. I believe it's not specific for MS Dynamics CRM, it could be verified with any site that is running IIS, with Windows authorization.

mholt · 2016-03-08T19:59:28Z

What would be interesting is lining up the fiddler output of trying both Caddy and what you know works.

shytikov · 2016-03-17T11:47:12Z

I can say that I failed all approaches to dig more information on the situation I'm facing. It seems to be problem is deeper than I thought.

And it seems to be that even caddy's closest rival — nginx has exactly the same issues, unless... you buy a special module from them, which actually allows passin-through NTLM to backend. Here is the link to module description: http://nginx.org/en/docs/http/ngx_http_upstream_module.html#ntlm

shytikov · 2016-03-17T11:55:02Z

Can someone help me to investigate and formalize the problem, so it would be easier to fix?

I believe it's something to do with TCP connections. Caddy re-instantiate connection to upsteam every time, and due to NTLM nature, these connections should be authenticated over and over again.

It should be a way how to prevent this type of behavior. For sure nginx and Fiddler2 can do that. The only thing I'm afraid of, will this approach compatible with HTTP/2... Hope it will...

mholt · 2016-03-18T00:28:14Z

I'm afraid I'm a bit in over my head here, knowing nothing about NTLM. Is NTLM an entirely different protocol over TCP from HTTP?

shytikov · 2016-03-18T08:15:49Z

@mholt please don't get me wrong :) I need some support in understanding caddy internals, not NTLM ones :) I'm quite new to Go through I was working with NTLM quite a white. Even on more or less low level.

I'm more afraid, that I will hit problems with HTTP/2 or will implement something that is strictly against standard :) And first thing that concerns me is it all possible to achieve at all?

abiosoft · 2016-03-18T08:26:27Z

Do you have a configured NTLM server anywhere publicly accessible that I can test with? If yes, then can you give pointers on what I should look for?

mholt · 2016-03-18T14:18:47Z

@shytikov I dunno the answer to your question of "Is it even possible" because I don't know anything about NLTM, that's what I mean. But I can point you to some places in Caddy's code if you want a look.
(Keep in mind I'm using permanent links here, actual code may have changed since.)

https://github.com/mholt/caddy/blob/6b801b111b1202a4e492ac8679f9255f62e9522d/middleware/proxy/proxy.go#L122 - Entry point of where a proxied request gets proxied to the backend.
https://github.com/mholt/caddy/blob/6b801b111b1202a4e492ac8679f9255f62e9522d/middleware/proxy/reverseproxy.go#L160-L175 - Headers are removed for the proxied request

jsimonetti · 2016-03-28T22:57:21Z

Afaik, when using NTLM, once the authentication of the client is done, the TCP connection needs to be kept alive. This means that any proxy will need to keep both its client facing connection and the server one alive.

shellster · 2016-04-02T22:59:23Z

@jsimonetti: Your statement is correct. Unlike Digest or Basic authentication where the authentication data is sent with each request, NTLM authentication authenticates the TCP connection. Once the connection is authenticated, future requests through that connection do not need to be authenticated (at least for a period of time). You can bypass this requirement, at a performance penalty, by doing the three-away authentication dance with each request, however, that final page request still has to be done over the same connection as the final authentication request. I am not familiar with how this issue is being handled since the advent of HTTP/2 or even if it is possible at this point in time. Obviously Caddy could hold open the connection on the backend and down grade to a HTTP/1.1 connection, but somehow you would still need to pass NTLM authentication headers to the user so that their browser could authenticate.

jsimonetti · 2016-04-03T11:23:56Z

@shellster All Windows authentication types break with http2

A Nice explaination of http2 on IIS can be found on this blogpost: http://blogs.iis.net/davidso/http2
Quote: "Windows authentication (NTLM/Kerberos/Negotiate) is not supported with HTTP/2. In this case IIS will fall back to HTTP/1.1."

mholt · 2017-02-21T05:26:26Z

Is this issue still relevant? I think this boils down to a feature request more than a bug, and now you can disable HTTP/2 on a per-site basis. Proxy can keep connections alive between requests. And Caddy's listeners can also be wrapped with plugins (#1349). Between all these things, maybe it's workable, I just don't know, and since it's been almost a year... I'm gonna close the issue for now. Feel free to continue discussion if you'd like, I just am not going to be able to do anything on it myself for now.

kringon · 2017-02-24T19:49:03Z

I'm looking into using caddy as a reverse proxy just for this case. If Caddy can svolve this, it is unique among the open source alternatives.

@mholt I'm going to research this with http/2 disable and keep-alive to see if it is duable vs reporting services on IIS that uses NTLM / Negotiation

mholt · 2017-02-24T19:52:35Z

We can reopen this issue if there is some movement that shows this is feasible. 👍

DrewRidley · 2019-06-14T00:23:07Z

I personally have interest in this becoming an option with a caddy reverse proxy!

mholt · 2019-06-14T03:56:42Z

@DrewDaPilot What is your use case? Would you be willing to contribute this feature to speed things up?

mholt · 2019-11-05T23:35:49Z

@shytikov @jsimonetti @shellster @kringon @DrewDaPilot

Got this working today. Caddy 2 can reverse proxy NTLM (at least, Windows Admin Center, I'm assuming it'll work for other stuff too).

Scroll down from here and you'll see how to use the http_ntlm transport module: https://github.com/caddyserver/caddy/wiki/v2:-Documentation#httphandlersreverse_proxy

Here's my config, for example:

{
	"handler": "reverse_proxy",
	"transport": {
		"protocol": "http_ntlm",
		"tls": {
			"insecure_skip_verify": true
		}
	},
	"upstreams": [
		{"dial": "wac:1080"}
	]
}

This is a commercial feature of nginx but it is open source in Caddy. Let me know how it goes!

alexbrina · 2020-05-26T12:17:54Z

I see a couple of commits removing NTLM support (since v2.0.0-rc2). Please confirm if it was removed, (or maybe moved somewhere else).

francislavoie · 2020-05-26T12:22:57Z

Yeah, it was removed from core and moved here as a plugin: https://github.com/caddyserver/ntlm-transport

For reference, the list of known Caddy v2 plugins is here for the time being: https://caddy.community/t/list-of-caddy-2-modules/7839

mholt added the help wanted 🆘 Extra attention is needed label Mar 21, 2016

mholt closed this as completed Feb 21, 2017

mholt added feature ⚙️ New feature or request and removed help wanted 🆘 Extra attention is needed labels Feb 21, 2017

mholt added this to the v2.0.0-beta10 milestone Nov 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reverse proxy on ASP.NET sites using NTLM authentication is not working #663

Reverse proxy on ASP.NET sites using NTLM authentication is not working #663

shytikov commented Mar 8, 2016

mholt commented Mar 8, 2016

shytikov commented Mar 8, 2016

mholt commented Mar 8, 2016

shytikov commented Mar 17, 2016

shytikov commented Mar 17, 2016

mholt commented Mar 18, 2016

shytikov commented Mar 18, 2016

abiosoft commented Mar 18, 2016

mholt commented Mar 18, 2016

jsimonetti commented Mar 28, 2016

shellster commented Apr 2, 2016

jsimonetti commented Apr 3, 2016

mholt commented Feb 21, 2017

kringon commented Feb 24, 2017

mholt commented Feb 24, 2017

DrewRidley commented Jun 14, 2019

mholt commented Jun 14, 2019

mholt commented Nov 5, 2019

alexbrina commented May 26, 2020

francislavoie commented May 26, 2020

Reverse proxy on ASP.NET sites using NTLM authentication is not working #663

Reverse proxy on ASP.NET sites using NTLM authentication is not working #663

Comments

shytikov commented Mar 8, 2016

1. What version of Caddy are you running (caddy -version)?

2. What are you trying to do?

3. What is your entire Caddyfile?

4. How did you run Caddy (give the full command and describe the execution environment)?

5. What did you expect to see?

6. What did you see instead (give full error messages and/or log)?

mholt commented Mar 8, 2016

shytikov commented Mar 8, 2016

mholt commented Mar 8, 2016

shytikov commented Mar 17, 2016

shytikov commented Mar 17, 2016

mholt commented Mar 18, 2016

shytikov commented Mar 18, 2016

abiosoft commented Mar 18, 2016

mholt commented Mar 18, 2016

jsimonetti commented Mar 28, 2016

shellster commented Apr 2, 2016

jsimonetti commented Apr 3, 2016

mholt commented Feb 21, 2017

kringon commented Feb 24, 2017

mholt commented Feb 24, 2017

DrewRidley commented Jun 14, 2019

mholt commented Jun 14, 2019

mholt commented Nov 5, 2019

alexbrina commented May 26, 2020

francislavoie commented May 26, 2020

1. What version of Caddy are you running (`caddy -version`)?