Existing HTTP/3 connections become unusable during config update #6917
Description
When a new config is pushed to caddy using POST http://localhost:2019, clients with existing HTTP/3 connections to the caddy server lose the ability to connect for a short period of time, causing active users to experience downtime.
I was able to reproduce this with a clean install on a new EC2 instance:
- Create a t4g.nano instance on EC2 with Ubuntu 24.04 (arm64).
- Open TCP ports 22, 80, 443 and UDP port 443.
- Install Caddy v2.9.1 for Linux arm64
- Create a
config.jsonfile on the server (replaceTESTDOMAINwith a domain you control):
{
"apps": {
"http": {
"servers": {
"srv0": {
"listen": [ ":443" ],
"routes": [
{
"match": [ { "host": [ "TESTDOMAIN" ] } ],
"handle": [
{
"handler": "subroute",
"routes": [
{
"group": "group1",
"handle": [
{
"body": "hello world",
"handler": "static_response"
}
]
}
]
}
],
"terminal": true
}
]
}
}
},
"tls": {
"automation": {
"policies": [
{
"subjects": [ "TESTDOMAIN" ]
}
]
}
}
}
}
- Add an
Arecord forTESTDOMAINthat matches the public IPv4 address of the EC2 instance. - Run
curl -d @config.json -H 'content-type: application/json' http://localhost:2019/loadon the server - Go to
https://TESTDOMAINin Google Chrome (tested with both Chrome for Android and desktop). - "hello world" should be shown.
- Refresh the page a bunch. It should load just fine.
- Edit
config.json, changing "hello world" to "hello world 2". - Run
sleep 3 && curl -d @config.json -H 'content-type: application/json' http://localhost:2019/loadon the server - Quickly switch to Google Chrome and start refreshing the page rapidly. The refreshes will work until
sleep 3finishes, at which point the page will hang and refuse to load. - Navigate to
https://TESTDOMAINfrom a different browser and observe that the page does in fact load. - Refresh Google Chrome again, and observe that the page still does not load.
- Wait 1-2 minutes.
- Refresh Google Chrome again. The page will now load correctly.
Blocking UDP port 443 seems to fix this problem.
Log output during config change:
{"level":"info","ts":1742505571.2683778,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_ip":"127.0.0.1","remote_port":"40602","headers":{"Accept":["*/*"],"Content-Length":["885"],"Content-Type":["application/json"],"User-Agent":["curl/8.5.0"]}}
{"level":"info","ts":1742505571.2690651,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
{"level":"info","ts":1742505571.269525,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1742505571.269545,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1742505571.2697465,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"warn","ts":1742505571.2697577,"msg":"quic listener tls configs are more than 2","number of configs":3}
{"level":"info","ts":1742505571.2697651,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"warn","ts":1742505571.269796,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"warn","ts":1742505571.2698002,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":80"}
{"level":"info","ts":1742505571.269803,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
{"level":"info","ts":1742505571.2698061,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["caddybugtest.timmclean.net"]}
{"level":"info","ts":1742505571.2698162,"logger":"http","msg":"servers shutting down with eternal grace period"}
{"level":"info","ts":1742505571.2710686,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
{"level":"info","ts":1742505571.2711222,"logger":"admin.api","msg":"load complete"}
{"level":"info","ts":1742505571.2733417,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}