httpserver: Configurable shutdown delay

[mholt]

This config setting allows scheduling a shutdown for the near future when a config is reloaded or the server is exiting. It blocks config reloads / signals while it waits and there is no cancelling it.

This is useful as it allows the server to continue operating normally before a config is unloaded. This delay happens before the grace period is initiated, meaning that new connections are still accepted during this time just like normal. After the delay completes, the grace period begins and new connections are no longer accepted and the graceful shutdown commences (idle connections closed, active connections closed once idle, etc).

Adds two new placeholders:

• {http.shutting_down} will return true if the server is shutting down, or false otherwise.
• {http.time_until_shutdown} will return the duration of the delay that remains if the server is scheduled to shut down. Otherwise it returns nil/empty.

This allows health check endpoints to announce that they will soon be going down so that this instance can be moved out of the rotation or a replacement instance can be spun up. For example:

{
	shutdown_delay 10s
}

...


	handle /health-check {
		@goingDown vars {http.shutting_down} true
		respond @goingDown "Bye-bye in {http.time_until_shutdown}" 503
		respond 200
	}

Note that this behavior is a little wonky for config reloads specifically. During a config change (as opposed to a config unload), two configs are running simultaneously for a time while the old one shuts down. Thus a request to a health check endpoint has the random chance of hitting the old server or the new one, since the grace period has not started yet. I'm trying to decide if that is correct behavior, or whether it should ONLY do this for a final shutdown (i.e. no new/replacement config).

@mxrlkn please try this out! :)

Closes #2843

[mholt]

This should probably only be enforced for listeners that are not being used by the next configuration (i.e. server is shutting down, not reloading). This can be determined by accessing the count in the usagepool. I think this is currently very unexported but I wonder if we can access it.

[francislavoie]

FWIW I think it would be useful to have a built-in way to check if a shutdown is actually for a reload or not. Right now, I think plugins need to set up their own UsagePool just for that purpose, which is a lot of awkward boilerplate for something that should just be a simple boolean check.

[mholt]

I'm currently refactoring this for a more proper implementation: the placeholders will be per-server (if I can figure that part out), since a config reload might stop some listeners but not others. If any listener is being stopped, then the delay will occur, otherwise it will not.

What's the significance of a config reload? All the clients should care about is whether the listener is going away or not.

[francislavoie]

For the case of the caddy-exec, it has startup and shutdown flags to indicate when a particular command should be run; you only want to start some attached server/process at startup and stop it on shutdown, no need to do that on every reload. https://github.com/abiosoft/caddy-exec

[mholt]

That's true, but this is only for HTTP server internals. Reload or shutdown will be irrelevant here (either way, it will report a shutdown if the listener is closing, which should be all that matters).

[mholt]

Ok, refactor complete, and I'm actually quite happy with things... much better than before IMO. Now shutdowns are announced per-server and whether it's a process exit or a reload is irrelevant.

This does, unfortunately, break the http3 package setting the Alt-Svc header, because the http3 server is reused for potentially multiple listeners, yet it requires a single hard-coded address or port in its struct. So I opened an issue upstream: quic-go/quic-go#3486

(Edit: No it doesn't, I just made a mistake... fixed it.)

[mholt]

I'm a dum dum. HTTP/3 works just fine with this. The error was in my code, I didn't realize something got messed up in the refactor. Sorry about that Marten! 😅 Thanks for being patient and rigorous.

I'll merge this in a moment.