Change in proxy config not reflected by in-process restart (http2 keep-alive connection not closed)

[tpng]

1. What version of Caddy are you running (caddy -version)?

master

2. What are you trying to do?

Reload Caddyfile with new proxy setting with USR1 using in-process restart.

3. What is your entire Caddyfile?

Before:

www.example.com (domain must be served on https, e.g. let's encrypt)

proxy / www.google.com {
        proxy_header Host {host}
        proxy_header X-Real-IP {remote}
        proxy_header X-Forwarded-For {remote}
        proxy_header X-Forwarded-Proto {scheme}
}

After:

www.example.com

proxy / www.yahoo.com {
        proxy_header Host {host}
        proxy_header X-Real-IP {remote}
        proxy_header X-Forwarded-For {remote}
        proxy_header X-Forwarded-Proto {scheme}
}

4. How did you run Caddy (give the full command and describe the execution environment)?

caddy

5. What did you expect to see?

After kill -USR1 on caddy pid, should show error page of Yahoo instead of Google.

6. What did you see instead (give full error messages and/or log)?

Still showing error page of Google.

7. How can someone who is starting from scratch reproduce this behavior as minimally as possible?

1. Start caddy with Caddyfile proxying Google
2. Change Caddyfile to proxy Yahoo
3. kill -USR1 CADDY_PID
4. Observe caddy still proxying Google

[tpng]

It seems on the first reload in the browser it will still proxy Google, but if I reload the page again it will proxy Yahoo.

@alexex
Can you please confirm if this is the same behavior you encounter in #806 ?

[alexanderjulo]

For me refreshing did not change anything as far as I remember, I would still encounter a 502, maybe because of the returned status code?

[tpng]

Ah, seems Firefox is caching the result.
With curl the proxy is returning the correct page after Caddy reload.

[tpng]

@alexex
Could you try to use curl in your case and see if the result is different?

[abiosoft]

I've used that signal couple of times and I do not have this issue. I hope its not a browser caching issue. You may want to try curl, multiple browsers and private windows.

[tpng]

It seems it is not a browser caching issue.

In my test case I setup 2 instances of caddy to be proxied, one listening on port 8081 and another on 8082.
If I change the Caddyfile (listening on https with Let's encrypt) to proxy 8082 instead of 8081, there are 2 different behaviors from browser and curl.

curl: the request is sent correctly to 8082 (request in log of 8082)

browser: in the same tab, disabling cache & reloading will still send request to 8081. (request in log of 8081)
Only if I open a new private window in Chrome will the request send correctly to 8082.
Private window in Firefox still send to 8081.

[tpng]

Since the issue only occurs with in-process restart, I guess it has something to do with keep-alive connections.

[mholt]

The reload can take up to 5 seconds (to kill keep-alive connections) -- did you wait 5 seconds before trying again?

(FWIW I cannot reproduce as long as I disable my browser cache.)

[alexanderjulo]

So I do not know why exactly but even after a few minutes while curl reliably returns the new proxy on chrome I still get a 502, even when I have the inspector open and disable caching there. While when I try it in an anonymous windows it seems to work. Something very funky seems to be going on here, but I have no idea what it is.

[tpng]

It seems the issue is only reproducible on http2 enabled servers. (served on https and with browser supporting http2).

[tpng]

With caddy -http2=false, Caddy behaves as @mholt stated, the correct site is proxied after 5 seconds.