Modify Caddy config using API called from proxied server

[anderspitman]

I'm working on a service that receives JSON requests and uses them to make changes to my Caddy config (for dynamically setting up proxies). So basically I'm publicly (with auth) exposing a wrapped subset of Caddy's own API.

However, I'm running into issues and I think it has something to do with the fact that my service is itself proxied by the Caddy instance it is modifying. If I try to POST a new Caddy route from within my service, it hangs for about 5 seconds, then the Caddy API request finally returns, but my original JSON client receives an empty response from Caddy. I'm guessing Caddy is killing the original request after some timeout.

Is there a proper way to accomplish what I'm trying to do?

[anderspitman]

I just confirmed that if I make the request directly to my proxied service, without going through Caddy, the request to the Caddy API returns instantly as expected. I indeed seem to be in some sort of a deadlock situation.

[mholt]

Yeah, that's exactly what's happening. Same thing happens if you try to set up a proxy to the admin endpoint within Caddy itself. The admin endpoint can't finish the config change if it's waiting for any outstanding requests to complete with the old config -- it will ask them to shut down of course, but in this case the request is not complete until the admin request is complete which is not request until the original request is complete, which ... you get the idea. After a while (5s by default), Caddy forcefully shuts down remaining connections so that those resources can be cleaned up. That's why your client hangs for 5 seconds and then has nothing in the response.

Basically, the admin API is hard-coded to always return after the config change completes so you can get an error or a success value. If it were not so, you would never know whether the change was successful or not.

This has come up before though. I don't think it's a good idea, but if we make the config change asynchronous (i.e. don't wait until the new config has been applied) and return a 202 status, then that should solve the deadlock. However, there's no way to know if your change was successful then.

[anderspitman]

Makes sense. Is there a realistic way to avoid ever killing old connections? ie keeping previous versions of the config around until they drain naturally? So new connections would go to the new config, but old connections can stay around longer.

Or what about adding a query param like async=true, which would cause the 202 you described. Then I could set an @id field on the config and perform a 2nd request to verify it matches what I expect before proceeding.

What are my options without requiring changes to Caddy? Run an SNI proxy and manage the certs for my admin service separately from Caddy? Or is there another higher level pattern I can use here to accomplish the same overall goal?

[francislavoie]

Would it be possible to respond to the incoming request before actually sending the request to Caddy? You could then cache the response and have the incoming client make another request ~5-6 seconds later to check for the result (or just poll until it gets a good response, with some timeout). That way the incoming connection wouldn't be killed off, because it would be closed before even talking to the Caddy admin API.

[anderspitman]

Hm that would work but feels even more hacky than my idea. Maybe Caddy isn't the right tool for what I'm trying to do? Are you aware of any reverse proxies that have a decent API and maintain old configs more aggressively?

[anderspitman]

What I'm building is going to need to support multi-tenant instances behind a single IP, each with their own Caddy managing the certs. So I'm going to need some sort of an SNI proxy anyway in that situation. So maybe I just map an admin service instance in addition to the Caddy instance for each tenant. I should be able to use certmagic in the admin service without too much trouble.

Any recommendations on a good SNI proxy? The most popular ones seem to be pretty oldschool C projects that aren't designed for dynamic configuration.

[francislavoie]

Why not just use Caddy as the SNI proxy itself? Are you aware of On-Demand TLS? https://caddyserver.com/docs/automatic-https#on-demand-tls

Caddy can work in a cluster if you have them share the same storage, so you can load balance to multiple Caddy instances if you need. https://caddyserver.com/docs/automatic-https#storage

[anderspitman]

Can Caddy do TCP passthrough by matching SNI names? My cursory poking around today didn't yield any obvious way to do that. My only real reason for not using Caddy is it's a bit heavy for that task. I prefer to use the simplest tool at each stage, and SNI proxying is a very simple process. But there's also value in reusing tools, so if Caddy can do it I'll definitely start with that.

I am aware of On-Demand TLS. Unfortunately it's not sufficient for my purposes because I don't know what port to proxy to until the admin request comes in.

Shared storage is an interesting option.

[francislavoie]

Project Conncept can do TCP proxying by SNI (sponsor-only plugin for now, until the sponsorship goal is met).

Unfortunately it's not sufficient for my purposes because I don't know what port to proxy to until the admin request comes in.

I'm thinking this could be something that a plugin provides, i.e. dynamic port picking as a middleware before proxying. It could fill a placeholder value that you could then use in the dial address of the proxy. That way, you wouldn't need to reload at all. Your middleware would just need to read from either some in-memory mapping or from file, or whatever. Might not be too difficult to implement: https://caddyserver.com/docs/extending-caddy

[mholt]

@anderspitman

Is there a realistic way to avoid ever killing old connections? ie keeping previous versions of the config around until they drain naturally? So new connections would go to the new config, but old connections can stay around longer.

For frequent config reloads (i.e. same order of magnitude of frequency as the duration of requests), this can lead to resource exhaustion pretty quickly.

Or what about adding a query param like async=true, which would cause the 202 you described. Then I could set an @id field on the config and perform a 2nd request to verify it matches what I expect before proceeding.

That is more or less what I had in mind when I wrote my comment, but, async makes everything more complicated.

Run an SNI proxy and manage the certs for my admin service separately from Caddy? Or is there another higher level pattern I can use here to accomplish the same overall goal?

Since you're wrapping/abstracting the API calls, you could make the API calls asynchronously so that the downstream requests can unblock. At least, until we figure out a better way to handle it.

Can Caddy do TCP passthrough by matching SNI names?

That's what Project Conncept is for. https://github.com/mholt/conncept -- currently accessible to sponsors until we reach our goal. 👍

[anderspitman]

Cool, I'll look into making a plugin and conncept as well. Should be able to hack together a solution out of everything discussed. Thanks!

[francislavoie]

Great, since there's nothing actionable for us, I'll close this now, but feel free to pop in for more discussion 👍

[mholt]

For future readers: Caddy 2.4 has remote admin features, so you can expose a secured socket to the admin API directly to a public interface, without needing to proxy it.