ERR_PROXY_CONNECTION_FAILED

[testcaoy7]

1. Is bug reproducible with latest forwardproxy build?

Yes

2. What are you trying to do?

Config a forward proxy.

3. What is your entire Caddyfile?

example.com:8443
log access.log
tls /root/cert/cert.cer /root/cert/privatekey.pem
forwardproxy {
        basicauth user1 12345678
        ports 10443
        hide_ip
        hide_via
        probe_resistance secret.localhost
        response_timeout 30
        dial_timeout 30
}

4. How is your client configured?

SwitchyOmega plug-in with Chromium browser.

5. How did you run Caddy? (give the full command and describe the execution environment). If multiple servers are used (for example with upstream), describe those as well.

caddy -conf CaddyFile

6. Please paste any relevant HTTP request(s) here.

7. What did you expect to see?

Browse Internet through a HTTPS proxy. (example.com:10443)

8. What did you see instead (give full error messages and/or log)?

Chromium reports “ERR_PROXY_CONNECTION_FAILED” and the access.log is empty.

9. How can someone who is starting from scratch reproduce the bug as minimally as possible?

[mdlenoir]

me too, website is all fine, but forwardproxy (lastest version from caddyserver.com) just not working...
example.com {
tls xxx@xxx.com {
curves p256
key_type p256
}

header / {
Strict-Transport-Security "max-age=31536000;"
X-XSS-Protection "1; mode=block"
X-Content-Type-Options "nosniff"
X-Frame-Options "DENY"
}
proxy /sehgew localhost:23231 {
websocket
transparent
}

proxy /dewew localhost:51234 {
    transparent
}

proxy / localhost:12345 {
    transparent
    except /xxxxxx
}

forwardproxy {
basicauth user1 12345678
hide_ip
hide_via
probe_resistance secret.localhost
}
gzip
log /var/log/caddy/access.log
errors /var/log/caddy/errors.log
}

And I tried simplest configure of caddyfile, still not working...
example.com {
tls xxx@xxx.com
forwardproxy {
basicauth user1 12345678
probe_resistance secret.localhost
}
}

[sergeyfrolov]

Confirming that I have reproduced the error and will look into. I am very busy right now, but the issue is serious, so I'll try to spend some time on it.

The issue seems to not occur, if I remove probe_resistance from Caddyfile.
@testcaoy7 and @mdlenoir, it would be helpful, if you could confirm that you also do not observe the issue with probe_resistance disabled. (I do not suggest disabling it permanently, just as a test)

[sergeyfrolov]

Specifically, It appears that forwardproxy is no longer able to see requests to *.localhost. So, if something.localhost is used as a probe_resistance link, forwardproxy won't be able to serve HTTP 407 response to user, causing switchyomega to not offer the credentials.

This could be caused by a change in either Chromium or Caddy.

Note that any domain could be used for probe_resistance; the advantage of .localhost addresses was that they will never be DNS resolved and HTTP request will never hit the network, if proxy is not in use, making it a mistake-proof mechanism for preserving the secrecy of the secret domain. You can try using other domain for probe_resistance.

[mdlenoir]

Hi, now I can see Hidden Proxy Page! without .localhost using Chrome and SwitchyOmega.
“Congratulations, you are successfully authenticated to the proxy! Go browse all the things!”
But proxy is not working, caddy error log says “[ERROR 0 ] Proxy-Authorization is required! Expected format: (type)(credentials)”.

If I remove probe_resistance, error logs “[ERROR 407 ] Invalid credentials“, not working...

[sergeyfrolov]

Yep, it's Chrome.
Firefox + SwitchyOmega will work with secret domains that end with .localhost.

TODO for myself before closing the issue: make changes in documentation of probe_resistance, and probably stop recommending .localhost domains.

I believe the issue is resolved, and could be closed; but going to wait for a while to see if @testcaoy7 would have time to report their experience with Firefox/non-localhost secret domain.

@mdlenoir, I am unable to reproduce the new issue you are having. It is likely that you got a typo when inputting credentials, or that browser remembered wrong/old credentials. Try recreating profile in SwitchyOmega/restart the browser.

[mholt]

Ouch, do we know what changed in Chrome?

I'm relieved that it's not an internal Caddy change/bug though!

[sergeyfrolov]

I would not call it a bug in either of those things, just an interesting gimmick, that we were taking advantage of. Previously, when user visited something.localhost domain:

• if proxy is in use: Chrome sends request for something.localhost to the proxy (the domain is encrypted /w TLS).
• if proxy not in use: Chrome does not send requests anywhere, does not DNS-resolve the something.localhost, etc.

As a result, if user accidentally by mistake entered their secret link, when proxy was not in use, the request would not hit the network, helping preserve the secret link. Not anymore.

If you use github.com or nonexistentdomain.whatever as a secret link, and accidentally type it into a browser, that is not configured to use proxy, the requests will hit the network, potentially leaking the secret link. I don't think censors are sophisticated enough to detect this, but it was nice to have it mistake-proof.

[mholt]

Looks like this is related: https://bugs.chromium.org/p/chromium/issues/detail?id=964231

Comment 1 says:

You need to use: --proxy-bypass-list="<-loopback>" when launching Chrome.

The proxy support in Chrome WIP doc has bypass rules for <local> and <-loopback>.

[sergeyfrolov]

@mholt many thanks for digging that up! This issue confirms that it was an intentional decision on Google Chrome's part.

I think it's easier to use other domains, than to pass flags to Chrome, since "the potential leak" isn't really a serious issue, especially when used with plausible domains.

[testcaoy7]

I am sorry. But I still got "ERR_PROXY_CONNECTION_FAILED" after the remove of "probe_resistance secret.localhost".

My server does not allow 80 and 443 port. So I ran forwardproxy at 10443 port. But forwardproxy doesn't accept my connection.

[sergeyfrolov]

@testcaoy7 @mdlenoir are you still getting "ERR_PROXY_CONNECTION_FAILED" after using non-localhost probe_resistance link?
I find that quite often the browser/SwitchyOmega plugin would cache some outdated (and not correct anymore) credentials/configuration. But after I delete and create again the SwitchyOmega profile, it works.

[testcaoy7]

@sergeyfrolov I just set a new server and create a new SwitchyOmega profile, it works.