reverseproxy: Support performing pre-check requests

[francislavoie]

So, this is super cool. We were just missing two small pieces plus a bug fix to make this pattern possible.

Recently, we found out that there's demand for integrating Caddy with Authelia authelia/authelia#1241, so that Authelia can be used for acting as an auth gateway for apps served by Caddy. The way it's typically done with other proxies is with a built-in feature called ForwardAuth in Traefik and auth_request in Nginx. TL;DR, an HTTP request is made to Authelia, and Authelia either responds with a 200 if the request authorized 👍 or with a 401 or redirect if auth is required 👎

So that got me thinking, reverse_proxy can make requests (obviously) and we've recently implemented a quite flexible handle_response feature that makes is quite simple to interact with the proxy response and even ignore the response body if we need. So what if we just use reverse_proxy to perform ForwardAuth-like functionality? That means we don't need a plugin, and it would work basically out-of-the-box with Caddy.

The key bits that were missing though, is that we need a way to tell the proxy "don't use the request body" and "always make GET requests" so that the request body isn't consumed so it can be actually used by a later HTTP handler. That's pretty easy, so I added no_body and override_method subdirectives to do this. We'd also need a way let reverse_proxy not be a terminal HTTP handler... but... turns out, handle_response was already implemented to work that way!

Then I started testing it a bit. Turns out that handle_response had a small bug, it would pass the cloned request to subsequent routes; that's bad, because then header_up manipulations would become permanent and apply to subsequent routes, and if we used no_body, we wouldn't have access to the body, etc. So I rewired things so that the original request is passed through subsequent routes. Fixed!

So here's how I tested this:

Edit: Note that this is no longer how the config looks, read further comments below for the changes

{
	debug
}

:8881 {
	log
	route {
		# pre-check request
		reverse_proxy :8882 {
			# setup, don't want to consume the body
			override_method GET
			no_body
			
			# just to prove that the later handlers
			# _don't_ see this header (see :8883)
			header_up Bar bar-header

			# handle the response, set a header on
			# the original request so subsequent
			# handlers can pass it through
			handle_response {
				request_header Foo foo-header
			}
		}

		# the "actual" handling, e.g. your app
		reverse_proxy :8883
	}
}

# pre-check, just responding with a 200 response
:8882 {
	log
	respond "We're good to go!"
}

# your app
:8883 {
	log
	respond "{header.Foo} {header.Bar}
{http.request.body}"
}

And then making a request like this; a POST to prove that :8882 indeed sees a GET, and :8883 sees the original POST, and the body:

$ curl -v http://localhost:8881 -H "Content-Type: application/json" -d '{"productId": 123456, "quantity": 100}'
*   Trying 127.0.0.1:8881...
* Connected to localhost (127.0.0.1) port 8881 (#0)
> POST / HTTP/1.1
> Host: localhost:8881
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 38
> 
* upload completely sent off: 38 out of 38 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Length: 51
< Date: Thu, 28 Apr 2022 03:37:48 GMT
< Server: Caddy
< Server: Caddy
< Content-Type: text/plain; charset=utf-8
< 
foo-header 
{"productId": 123456, "quantity": 100}
* Connection #0 to host localhost left intact

And the logs from this:

2022/04/28 04:22:21.924	DEBUG	http.handlers.reverse_proxy	selected upstream	{"dial": ":8882", "total_upstreams": 1}
2022/04/28 04:22:21.925	INFO	http.log.access	handled request	{"request": {"remote_ip": "127.0.0.1", "remote_port": "39378", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8881", "uri": "/", "headers": {"Accept": ["*/*"], "Bar": ["bar-header"], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["http"], "User-Agent": ["curl/7.74.0"], "Content-Type": ["application/json"], "X-Forwarded-Host": ["localhost"], "Accept-Encoding": ["gzip"]}}, "user_id": "", "duration": 0.000050308, "size": 17, "status": 200, "resp_headers": {"Server": ["Caddy"], "Content-Type": []}}
2022/04/28 04:22:21.925	DEBUG	http.handlers.reverse_proxy	upstream roundtrip	{"upstream": ":8882", "duration": 0.000680542, "request": {"remote_ip": "127.0.0.1", "remote_port": "41778", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8881", "uri": "/", "headers": {"Content-Length": ["38"], "Content-Type": ["application/json"], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Host": ["localhost"], "User-Agent": ["curl/7.74.0"], "Accept": ["*/*"], "Bar": ["bar-header"]}}, "headers": {"Server": ["Caddy"], "Date": ["Thu, 28 Apr 2022 04:22:21 GMT"], "Content-Length": ["17"]}, "status": 200}
2022/04/28 04:22:21.925	DEBUG	http.handlers.reverse_proxy	handling response	{"handler": 0}
2022/04/28 04:22:21.925	DEBUG	http.handlers.reverse_proxy	selected upstream	{"dial": ":8883", "total_upstreams": 1}
2022/04/28 04:22:21.925	INFO	http.log.access	handled request	{"request": {"remote_ip": "127.0.0.1", "remote_port": "35452", "proto": "HTTP/1.1", "method": "POST", "host": "localhost:8881", "uri": "/", "headers": {"Content-Type": ["application/json"], "Foo": ["foo-header"], "X-Forwarded-For": ["127.0.0.1"], "Accept-Encoding": ["gzip"], "User-Agent": ["curl/7.74.0"], "Content-Length": ["38"], "Accept": ["*/*"], "X-Forwarded-Host": ["localhost"], "X-Forwarded-Proto": ["http"]}}, "user_id": "", "duration": 0.00009696, "size": 50, "status": 200, "resp_headers": {"Server": ["Caddy"], "Content-Type": []}}
2022/04/28 04:22:21.925	DEBUG	http.handlers.reverse_proxy	upstream roundtrip	{"upstream": ":8883", "duration": 0.000544203, "request": {"remote_ip": "127.0.0.1", "remote_port": "41778", "proto": "HTTP/1.1", "method": "POST", "host": "localhost:8881", "uri": "/", "headers": {"X-Forwarded-Host": ["localhost"], "User-Agent": ["curl/7.74.0"], "Accept": ["*/*"], "Content-Type": ["application/json"], "Content-Length": ["38"], "Foo": ["foo-header"], "X-Forwarded-For": ["127.0.0.1"], "X-Forwarded-Proto": ["http"]}}, "headers": {"Server": ["Caddy"], "Date": ["Thu, 28 Apr 2022 04:22:21 GMT"], "Content-Length": ["50"]}, "status": 200}
2022/04/28 04:22:21.926	INFO	http.log.access	handled request	{"request": {"remote_ip": "127.0.0.1", "remote_port": "41778", "proto": "HTTP/1.1", "method": "POST", "host": "localhost:8881", "uri": "/", "headers": {"Content-Type": ["application/json"], "Content-Length": ["38"], "User-Agent": ["curl/7.74.0"], "Accept": ["*/*"]}}, "user_id": "", "duration": 0.001739929, "size": 50, "status": 200, "resp_headers": {"Server": ["Caddy", "Caddy", "Caddy"], "Date": ["Thu, 28 Apr 2022 04:22:21 GMT", "Thu, 28 Apr 2022 04:22:21 GMT"], "Content-Length": ["50"]}}

So, the logs in order:

• Selected :8882 as the upstream for the pre-check
• The :8882 server logs the request, notice it's a GET here and there's no Content-Length header
• The :8882 proxy logs its roundtrip
• The handle_response is selected and runs, then continues the handling chain
• Selected :8883 as the upstream for the "actual" handling of the request
• The :8883 server logs the request, notice it's a POST here and Content-Length is 38 (my dumb little JSON payload)
• The :8883 proxy logs its roundtrip
• The :8881 server finally logs the request and the 50 byte response (the Foo header value and echoed request body)

I just noticed as I write this though that the last log line has Server: Caddy three times 🤔 there might be a bug with the response writer, I'll need to look into this more closely to see what's going on, but interestingly the response in curl only has two (which is correct, i.e. :8881 and :8883 ultimately should be the only ones manipulating the response).

So with all this out of the way, this means that ForwardAuth can be done purely with reverse_proxy. But obviously this is pretty verbose and has a lot of boilerplate, so we'll probably provide a forward_auth Caddyfile directive, similarly to php_fastcgi which is a shortcut/sugar over reverse_proxy to make it nicer to use, with good defaults.

[james-d-elliott]

Hey I'm glad we managed to get this far. I didn't get time to test it for you tonight, however I will over the weekend. I must say it's been a pleasure working with you thus far. I can't wait to add this to Authelia's list of supported proxies.

[mholt]

This LGTM overall. I like the possibilities here. Interesting implicit "pass_thru" mode, like you mentioned in Slack.

[francislavoie]

Alright I did a bit more thinking and I pivoted a bit on how this will be configured. Instead of two new fields override_method and no_body, I think it's more elegant to embed a rewrite right into reverse_proxy. This gives us both method rewriting (the rewrite module has it built-in), and we get URI rewriting, which is important for the ForwardAuth pattern.

So the proposed config for ForwardAuth looks like this (long way with reverse_proxy; I didn't work on the forward_auth shortcut yet)

# the entrypoint
:8881 {
	log
	route {
		# pre-check request
		reverse_proxy :8882 {
			# rewrite to the auth endpoint, prevent body from being sent
			method GET
			rewrite "/api/verify?rd={scheme}://{hostport}{uri}"

			# send additional information about the original request
			header_up X-Forwarded-Method {method}
			header_up X-Forwarded-Uri {uri}

			# handle the response; copy a response header onto the
			# incoming request to let is pass through to the app
			@good status 200
			handle_response @good {
				request_header Remote-User {http.reverse_proxy.header.Remote-User}
				# Add more headers to copy here
			}

			# not authenticated; triggers a redirect to the auth gateway
			@needs-auth status 4xx
			handle_response @needs-auth {
				redir * {http.reverse_proxy.header.Location}
			}

			# oops, we got an error; terminate
			@bad status 5xx
			handle_response @bad {
				error {http.reverse_proxy.status_text} {http.reverse_proxy.status_code}
			}
		}

		# the "actual" handling, e.g. your app
		reverse_proxy :8883
	}
}

# pre-check, just responding with a 200 response
:8882 {
	log
	header Remote-User bob
	respond "Bob is logged in!"
}

# your app
:8883 {
	log
	respond "User: {header.Remote-User} | Body: {http.request.body}"
}

So the new thing is this:

method GET
rewrite /api/verify?rd={scheme}://{hostport}{uri}

This applies a rewrite to the cloned request (not the original request), changing both the method and the URI (path + query). Also, if the method is set to GET or HEAD, the body will implicitly be turned off, because bodies don't make sense for those methods and we need it turned off if we override probably so the body can be consumed by a later handler.

[francislavoie]

While testing, we realized there's a usecase where using the redir directive with 401 redirects, and possibly placeholders, so I made the Caddyfile adapter for redir a bit more permissive (JSON allowed it already cause it's just a static_response handler under the hood)

[james-d-elliott]

This is now confirmed as working with Authelia. We've not done thorough integration tests yet, but we will probably do this in the coming days. Here is my Caddyfile I used for testing:

Caddyfile

auth.example.com {
	tls /etc/caddy/pub.chain.pem /etc/caddy/priv.pem
	log
	reverse_proxy authelia:9091
}

app1.example.com {
	tls /etc/caddy/pub.chain.pem /etc/caddy/priv.pem
	log
	route {
		reverse_proxy authelia:9091 {
			method GET
			rewrite "/api/verify?rd=https://auth.example.com"

			header_up X-Forwarded-Method {method}
			header_up X-Forwarded-Uri {uri}

			@good status 200
			@bad status 5xx

			@needs-auth {
				header Location *
				status 302 303 401
			}

			handle_response @good {
				request_header Remote-User {http.reverse_proxy.header.Remote-User}
				request_header Remote-Groups {http.reverse_proxy.header.Remote-Groups}
				# Add more headers to copy here
			}

			handle_response @needs-auth {
				respond "" {http.reverse_proxy.status_code}
				header Location {http.reverse_proxy.header.Location}
			}

			handle_response @bad {
				# copy some headers?
				error {http.reverse_proxy.status_text} {http.reverse_proxy.status_code}
			}
		}

		respond "Application: app1. User: {header.Remote-User}. Groups: {header.Remote-Groups}. Body: {http.request.body}."
	}
}

app2.example.com {
	tls /etc/caddy/pub.chain.pem /etc/caddy/priv.pem
	log
	route {
		reverse_proxy authelia:9091 {
			method GET
			rewrite "/api/verify?rd=https://auth.example.com"

			header_up X-Forwarded-Method {method}
			header_up X-Forwarded-Uri {uri}

			@good status 200
			@bad status 5xx

			@needs-auth {
				header Location *
				status 302 303 401
			}

			handle_response @good {
				request_header Remote-User {http.reverse_proxy.header.Remote-User}
				request_header Remote-Groups {http.reverse_proxy.header.Remote-Groups}
				# Add more headers to copy here
			}

			handle_response @needs-auth {
				respond {http.reverse_proxy.status_code}
				header Location {http.reverse_proxy.header.Location}
			}

			handle_response @bad {
				# copy some headers?
				error {http.reverse_proxy.status_text} {http.reverse_proxy.status_code}
			}
		}

		respond "Application: app1. User: {header.Remote-User}. Groups: {header.Remote-Groups}. Body: {http.request.body}."
	}
}

A much simpler one is provided below that more closely represents other implementations which makes migration so much easier!

[james-d-elliott]

Just a small update to above, the following Caddyfile I believe more closely represents other implementations of this flow, and is much simpler:

Caddyfile

auth.example.com {
	tls /etc/caddy/pub.chain.pem /etc/caddy/priv.pem
	log
	reverse_proxy authelia:9091
}

app1.example.com {
	tls /etc/caddy/pub.chain.pem /etc/caddy/priv.pem
	log
	route {
		reverse_proxy authelia:9091 {
			method GET
			rewrite "/api/verify?rd=https://auth.example.com"

			header_up X-Forwarded-Method {method}
			header_up X-Forwarded-Uri {uri}

                        ## If the auth request:
                        ##   1. Responds with a status code IN the 200-299 range.
                        ## Then:
                        ##   1. Proxy the request to the backend.
                        ##   2. Copy the relevant headers from the auth request and provide them to the backend.
                        @good status 2xx
			handle_response @good {
                                request_header Remote-User {http.reverse_proxy.header.Remote-User}
                                request_header Remote-Groups {http.reverse_proxy.header.Remote-Groups}
                                request_header Remote-Name {http.reverse_proxy.header.Remote-Name}
                                request_header Remote-Email {http.reverse_proxy.header.Remote-Email}
			        # Add more headers to copy here
			}

                        ## If the auth request:
                        ##   1. Responds with a status code NOT IN the 200-299 range.
                        ## Then:
                        ##   1. Respond with the status code of the auth request.
                        ##   1. Copy the response except for several headers.
			@denied {
                                status 1xx 3xx 4xx 5xx
			}
			handle_response @denied {
				copy_response
                                copy_response_headers {
                                        exclude Connection Keep-Alive Te Trailers Transfer-Encoding Upgrade
                                }
			}
		}

		respond "Application: app1. User: {header.Remote-User}. Groups: {header.Remote-Groups}. Body: {http.request.body}."
	}
}

[francislavoie]

I've implemented the forward_auth shortcut directive:

auth.example.com {
	reverse_proxy authelia:9091
}

app1.example.com {
	forward_auth authelia:9091 {
		uri /api/verify?rd=https://auth.example.com
		copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
	}

	respond "Application: app1. User: {header.Remote-User}. Groups: {header.Remote-Groups}. Body: {http.request.body}."
}

This results in essentially the same config as @james-d-elliott posted in the previous comment.

Aside from these two subdirectives, uri and copy_headers, any other reverse_proxy subdirectives can be used, like load balancing, health checks, dynamic upstreams, tls config, etc.

[mholt]

This is looking great. v2.5.1 will be the most exciting patch release we've ever done 😋

[mholt]

LGTM. Well done! This is a neat feature and I'm impressed with how quickly and easily it came together. Thanks for your help, @james-d-elliott, and for your quality work as usual, @francislavoie.

[codecat]

I really like this approach, but am wondering if there's any possible bottleneck here when there's a large amount of requests - every 1 (authenticated) request would turn into 2 requests, wouldn't it? Even if it is routed internally on localhost, it probably has some overhead.

Has there been any testing into how performant this is?

[francislavoie]

Hey @codecat! 👋

This is the same approach as Traefik and Nginx have used; ideally the auth gateway should be on the same machine or close to, so it should only have like 1-2ms latency ideally, especially if it just checks a Cookie or something to verify that auth is good.

But @james-d-elliott has more experience with this. What kind of round-trip time do you typically see with Authelia?

[james-d-elliott]

So I actually have not benchmarked the difference, however that's because I have never personally felt like there was a drop in my personal experience. I had in mind doing some benchmarking in general because I believe we're one of the most efficient solutions available at present (15-20mb RAM, 0.00%-0.01% CPU consistently). I recall checking our policy engine and with 100 policies getting the last policy took about 4ms if I recall correctly. I'll add this idea to my list of things to benchmark.

Authelia is actually unofficially used by some fairly large companies too which I can't name the names of, some of which have in the 100's of thousands of employees; I don't really have any data on how many active users they have or how they've deployed it, but I plan to ask if they'd be willing to do anonymous write up to put on our site.

[codecat]

I'd love to see any data on that!

[helmut72]

Thank you for this great feature!

But one question. When I want to configure the same in Caddy like this, it doesn't work:
https://community.getgrist.com/t/a-template-for-self-hosting-grist-with-traefik-and-docker-compose/856

My configuration also catch /auth/login, but after login with Authelia, all Remote-* headers are empty when opening /.

Authelia use the FQDN:

    - domain: grist.example.com
      policy: one_factor

Snippet of my Caddyfile:

grist.example.com:443 {
	import wildcard
	import client_headers
	import log_formatted INFO

	redir /sign-out https://auth.example.com/logout?rd=https://grist.example.com

	forward_auth /auth/login authelia:9091 {
		uri /api/verify?rd=https://auth.example.com
		copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
	}

	reverse_proxy http://172.17.1.1:8000 {
		header_up Remote-Email {header.Remote-Email}
	        header_up Remote-Groups {header.Remote-Groups}
	        header_up Remote-Name {header.Remote-Name}
	        header_up Remote-User {header.Remote-User}
	}
}

[james-d-elliott]

My configuration also catch /auth/login, but after login with Authelia, all Remote-* headers are empty when opening /.

It shoud just be reverse_proxy http://172.17.1.1:8000, remove the block with all the header-up sections. Also the backend will receive the headers, not the browser (they are injected request headers).

[helmut72]

Makes no difference. header_up was just for testing and was my idea to force it. I have tried a lot yesterday with whoami image. Problem is, that the Remote-* headers are empty or not there at all when I request something else than /auth/login. If Remote-* headers are not filled, Grist will use a default user.
It works as expected if I enable forward_auth for the whole site, but then no public sharing is possible.

[francislavoie]

The only way that the headers can get filled is if forward_auth runs on every request where the headers are needed, because that's how the headers are fetched from Authelia. You'll need to change your request matcher to not only run on /auth/login.

All that said, I suggest we move this discussion to the forums; this isn't the right place for support at this point. https://caddy.community

[helmut72]

Yes, of course.

[james-d-elliott]

Edit: Sorry just saw the comment about moving the discussion.

As per this comment whoami is showing the expected headers, i.e. it's 100% working: authelia/authelia#3319 (comment)

[otbutz]

@francislavoie i can't find forward_auth in the docs?

[james-d-elliott]

caddyserver/website#228

[mikesutton]

This is super exciting - absolute well done to @francislavoie and @james-d-elliott .
I have a question - is the assumption that it works with other similar authentication tools like oauth2-proxy or basically are you saying authelia is the way to go?

This is not ideal (as KeyCloak) is the tool we use for our authentication, and I know Authelia does not currently support OIDC providers.

Push comes to shove, I'll put Authelia in place and use this new support to secure the internal apps.

[james-d-elliott]

It has not been specifically tested with anything other than Authelia. But the intention during implementation was not to reinvent the wheel as creating something for the sake of one product makes no sense since there are multiple similar implementations of what traefik call forward auth we made it operate theoretically nearly identical to that. So in short, yes, it should work. If you find an issue after making an earnest attempt I'm sure the caddy maintainers would be happy for you to ask on the forums (or potentially in an issue here if you find a legitimate bug).

I know Authelia does not currently support OIDC providers.

It acts as a provider, but yes, it does not support acting as a relying party as of yet.

[mikesutton]

Superb - making an earnest effort to make it work with oauth2 and Authelia as a fallback.
Thanks again.

[francislavoie]

@mikesutton my understanding is that keycloak doesn't support the forward auth flow, but that there is a workaround by running another piece of software like https://github.com/mesosphere/traefik-forward-auth which despite the name should work fine with Caddy as well (because the flow is the same).

Also, it has been tested by some Tailscale users and it works with their recent "nginx auth" https://tailscale.com/blog/tailscale-auth-nginx/ see #4763

[NakaTheShifu]

I've implemented the forward_auth shortcut directive:

auth.example.com {
	reverse_proxy authelia:9091
}

app1.example.com {
	forward_auth authelia:9091 {
		uri /api/verify?rd=https://auth.example.com
		copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
	}

	respond "Application: app1. User: {header.Remote-User}. Groups: {header.Remote-Groups}. Body: {http.request.body}."
}

This results in essentially the same config as @james-d-elliott posted in the previous comment.

Aside from these two subdirectives, uri and copy_headers, any other reverse_proxy subdirectives can be used, like load balancing, health checks, dynamic upstreams, tls config, etc.

Would this work for usage with goauthentik?

[francislavoie]

@NakaTheShifu yes, I think it should! See authentik's forward auth docs: https://goauthentik.io/docs/providers/proxy/forward_auth. If you have success with it, be sure to propose that Caddy is added to their docs!

[leoil]

Hi! I have a question regarding the behavior of subsequent requests after successful authentication (access is granted). Do these subsequent requests still need to pass through the authentication node, or is authentication bypassed for authenticated users, such as using a sessionToken?

[francislavoie]

Authentication is checked independently for every request. There's no state kept in Caddy between requests for this. So the auth upstream should write a cookie to avoid repeating auth on subsequent requests.