Flexibility with PROXY protocol header on the same port? #2450
Comments
That would be insecure, because if a client knows that the host won't add the PROXY header for a given connection, but also knows that the host is generally considered to be trustworthy, then that client could just send the PROXY header itself at the beginning of the connection. |
Sorry I don't follow? With a reverse proxy like Caddy or Traefik which does allows this flexibility, how is this insecure? Original responseJust so I don't misunderstand you, this is further clarification for me:
Connection flow:
Meanwhile:
Where is the security issue? If you already trust a host to receive PROXY headers from, then what is the consequence of accepting traffic that is not presenting that header? I think given that the receiving end disregards any PROXY header unless trusted, makes your concern invalid? Perhaps there are other factors related to a deployment that cause the conflicting opinion on security here? 🤷♂️ I've provided an example below that's quite easy to run with Please point out how this is insecure:
|
|
Here is a reproduction environment with both Traefik and Caddy involved. See the curl commands at the end for various scenarios tested. # compose.yaml
# Usage:
# - Bring up services: docker compose up -d --force-recreate
# - Run curl tests: docker compose run --rm client
services:
reverse-proxy:
image: docker.io/traefik:3.0
hostname: traefik.internal.test
networks:
default:
ipv4_address: 172.16.42.10
command:
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --providers.docker.network=dms-test-network
# Traefik will redirect any incoming HTTP requests to HTTPS:
- --entrypoints.web.address=:80
- --entrypoints.web.http.redirections.entryPoint.to=websecure
- --entrypoints.web.http.redirections.entryPoint.scheme=https
# Equivalent to Caddy `servers.trusted_proxies` for accepting `X-Forwaded-*` headers:
# https://doc.traefik.io/traefik/routing/entrypoints/#forwarded-headers
# NOTE: This is technically always enabled for Layer 4 (TCP) router/services.
# NOTE: Both entrypoints would need to be enabled if requests starts with HTTP/80,
# as either entrypoint would strip those headers otherwise.
#- --entryPoints.web.forwardedHeaders.insecure=true
#- --entryPoints.websecure.forwardedHeaders.insecure=true
- --entryPoints.websecure.address=:443
# CAUTION: Production usage should configure socket access better (see Traefik docs)
volumes:
- /var/run/docker.sock:/var/run/docker.sock
ports:
- "80:80"
- "443:443"
web:
# Can use this instead of build once Caddy 2.8 is released:
#image: caddy:2.8
hostname: caddy.internal.test
networks:
default:
ipv4_address: 172.16.42.20
# Caddy 2.8 makes PROXY protocol header optional on
# the receiving port, just like Traefik.
# Build master branch for unreleased Caddy 2.8:
build:
dockerfile_inline: |
FROM caddy:2.7-builder AS builder
RUN xcaddy build master
FROM caddy:2.7
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
labels:
- traefik.enable=true
# Only TCP services can enable PROXY Protocol:
# TLS passthrough delegates TLS termination to Caddy instead:
- traefik.tcp.routers.websecure.rule=HostSNI(`*`)
- traefik.tcp.routers.websecure.tls.passthrough=true
- traefik.tcp.routers.websecure.entrypoints=websecure
- traefik.tcp.routers.websecure.service=with-proxy
# Route to port 443 with PROXY protocol, Caddy will handle TLS:
- traefik.tcp.services.with-proxy.loadbalancer.server.port=443
- traefik.tcp.services.with-proxy.loadbalancer.proxyProtocol.version=2
# Skips PROXY protocol connection for https://no-proxy-header.test
- traefik.tcp.routers.no-proxy-header.rule=HostSNI(`no-proxy-header.test`)
- traefik.tcp.routers.no-proxy-header.tls.passthrough=true
- traefik.tcp.routers.no-proxy-header.entrypoints=websecure
- traefik.tcp.routers.no-proxy-header.service=without-proxy
# Route to port 443, Caddy will handle TLS:
- traefik.tcp.services.without-proxy.loadbalancer.server.port=443
# Skips PROXY protocol connection for https://https-router.test
# Traefik will terminate TLS instead of delegating to Caddy (no passthrough).
# Thus the Traefik cert is presented instead, and decrypted traffic is sent to Caddy.
- traefik.http.routers.no-proxy-header.rule=Host(`traefik-https-router.test`)
- traefik.http.routers.no-proxy-header.tls=true
- traefik.http.routers.no-proxy-header.entrypoints=websecure
- traefik.http.routers.no-proxy-header.service=without-proxy
# Route to port 80, Traefik handled TLS:
# NOTE: Caddy won't issue an HTTPS redirect for this host like it would others.
- traefik.http.services.without-proxy.loadbalancer.server.port=80
configs:
- source: caddyfile
target: /etc/caddy/Caddyfile
# Only intended for running via `docker compose run --rm -it client`
client:
hostname: client.internal.test
# An easy to recognize IP in the logs:
networks:
default:
ipv4_address: 172.16.42.42
# Prevent starting this service by default with `docker compose up`:
profiles:
- testing
build:
dockerfile_inline: |
FROM alpine
RUN apk add curl
command: ash /tmp/run.sh
configs:
- source: curl-cmds
target: /tmp/run.sh
networks:
default:
name: dms-test-network
ipam:
config:
- subnet: "172.16.42.0/24"
configs:
caddyfile:
content: |
# Global options:
{
local_certs
# Enable ProxyProtocol for incoming Traefik connections:
# https://caddyserver.com/docs/caddyfile/options#listener-wrappers
servers {
#log_credentials
# Optionally trust HTTP headers from Traefik (regardless of TCP/HTTP router):
trusted_proxies static 172.16.42.10/32
listener_wrappers {
# Must come before `tls`.
proxy_protocol {
# Only trust PROXY protocol connections from Traefik:
allow 172.16.42.10/32
}
tls
}
}
}
# These docs are mostly applicable to HTTP routed traffic:
# NOTE: TCP routers by Traefik do not understand HTTP traffic is handled,
# Thus any headers included (eg: X-Forwarded-For) are not discarded.
# https://caddyserver.com/docs/caddyfile/options#client-ip-headers
# > Pairing with trusted_proxies, allows configuring which headers to use to determine the client's IP address.
# > By default, only X-Forwarded-For is considered.
# https://caddyserver.com/docs/caddyfile/matchers#client-ip
# > Only requests from trusted proxies will have their client IP parsed at the start of the request;
# > untrusted requests will use the remote IP address of the immediate peer.
# https://caddyserver.com/docs/caddyfile/matchers#remote-ip
# > the first IP in the X-Forwarded-For request header, if present, will be preferred as the reference IP,
# > rather than the immediate peer's IP, which is the default.
(respond_with_details) {
log
# Requested Host => The HTTP `Host` header
# Remote Host IP => The host connecting directly to Caddy (eg: Traefik)
# Client IP => The real client IP (requires PROXY protocol or direct connection)
# ---
# NOTE: Traefik only provides these headers for connections using HTTP router/services not TCP:
# X-Forwarded-Host => Requested host that Caddy will match to a site block
# X-Forwarded-Server => Remote Host
# X-Forwarded-For => Client IP (Caddy will use this if `servers.trusted_proxies` is configured)
# X-Real-Ip => Client IP (Extra header from Traefik)
respond <<EOF
Caddy Received:
Requested Host: {host}
Remote Host IP: {remote_host}
Client IP: {client_ip}
---
X-Forwarded-Host: {header.X-Forwarded-Host}
X-Forwarded-Server: {header.X-Forwarded-Server}
X-Forwarded-For: {header.X-Forwarded-For}
X-Real-Ip: {header.X-Real-Ip}
EOF
}
example.test {
import respond_with_details
}
no-proxy-header.test {
import respond_with_details
}
http://traefik-https-router.test {
import respond_with_details
}
curl-cmds:
content: |
#!/bin/sh
# Curl options:
# - `--insecure` (`-k`) to ignore local generated certs (Caddy/Traefik) fail to verify (CA not in trust store)
# - `--location` (`-L`) to follow redirect for HTTP => HTTPS (used for direct Caddy example.test)
# - `--header` (`-H`) add/modify request headers sent.
# - `-w '\n'` for ensuring a final LF which is not part of the Caddy respond directive
# - `--connect-to` routes traffic to the Traefik reverse proxy:
# https://curl.se/docs/manpage.html#--connect-to
# TCP TLS router/service, with proxy header:
# Both remote and client IP are for the Client (remote probably should remain as Traefik IP?):
echo -e '\n- Client => Traefik (PROXY protocol) => Caddy (http://example.test)'
curl -L -H "X-Forwarded-For: 1.2.3.4" --insecure --connect-to ::traefik.internal.test http://example.test -w '\n'
# TCP TLS router/service, without proxy header:
# Both remote and client IP are for Traefik (expected):
# WARNING: With Caddy `servers.trusted_proxies` the curl client can add headers to fool Caddy here,
# as routing Layer 4 traffic in Traefik is not concerned with Layer 7 (HTTP).
# However when the PROXY protocol is used (like prior curl request), it has precedence for setting Client IP.
echo -e '\n- Client => Traefik (no PROXY protocol) => Caddy (http://no-proxy-header.test)'
curl -L -H 'X-Forwarded-For: 1.2.3.4' --insecure --connect-to ::traefik.internal.test http://no-proxy-header.test -w '\n'
# HTTP TLS router/service:
# Caddy `servers.trusted_proxies` must be enabled to resolve Client IP correctly here (expected), otherwise it will represent the Traefik IP.
# As the request is via an HTTP router the curl header is ignored - unless allowing the Traefik entrypoint to forward these headers from clients.
echo -e '\n- Client => Traefik (no PROXY protocol) => Caddy (http://traefik-https-router.test)'
curl -L -H 'X-Forwarded-For: 1.2.3.4' --insecure --connect-to ::traefik.internal.test http://traefik-https-router.test -w '\n'
echo -e '\n------\n'
# All three of these curl requests respond with remote and client IP as that of the client (which is correct):
echo -e '\n- Client => Caddy (http://example.test)'
curl -L --insecure --connect-to ::caddy.internal.test http://example.test -w '\n'
echo -e '\n- Client => Caddy (http://no-proxy-header.test)'
curl -L --insecure --connect-to ::caddy.internal.test http://no-proxy-header.test -w '\n'
# Internally to Caddy this endpoint doesn't offer HTTPS:
echo -e '\n- Client => Caddy (http://traefik-https-router.test)'
curl --connect-to ::caddy.internal.test http://traefik-https-router.test -w '\n'Output from running the two commands at the start of that config file: I think you are referring to
Caddy could just as easily not trust those headers via So the question is:
Given that:
Do you really need to mandate separate ports? Isn't that achieving the same as a trust policy handled on the same port? |
|
I'm afraid your follow-up question is fairly verbose and I'm not sure I follow your entire line of thought. I also don't fully understand Traefik's or Caddy's configuration, as I've used neither before.
I believe this is the case I'm referring to, yes: Traefik is not configured to send a PROXY header, while Caddy is configured to optionally (!) accept a PROXY header from Traefik. Now consider the following (which might or might not actually work depending on Traefik's configuration): If Traefik is acting as a dumb TCP forwarder without performing any protocol validation (i.e. checking whether protocol is valid HTTP), then it will pass the faked PROXY header from the client as-is to Caddy. Caddy will see the trusted IP address from Traefik and thus will trust the PROXY header, even if it was not generated by Traefik!
I believe this is what I mean by “dumb TCP forwarder”.
Because it's the only reliable way to accurately determine whether
I did not write that part of the specification, but there is an important difference between the two requirements:
I think that the paragraph in the protocol specification could be phrased more clearly, but I believe the MUST + SHOULD is sound.
Conditionally handling which hosts can connect without PROXY procotol is fine, if the decision is made without looking into the contents. In other words: It must be clear at connection time whether the client is expected to send a PROXY header or not. It must not be decided by checking whether it looks like the client sent a PROXY header or not. If no PROXY header is provided, but the client is trusted to expected one, the connection MUST be rejected. In fact HAProxy supports such a configuration with the
A trust policy on the same port is fine, if it's a decision between “client MUST send a PROXY header” and “no special handling is performed”. Making the PROXY header optional is not fine. Note the important distinction between “no special handling” and ”optional”. Hope that helps and is sufficiently clear. |
Apologies. I lack the time to be terse 😬 (but I've tried to briefly summarize)
It was quite helpful, thanks! However I remain at the same stance on the specification being too strict with it's wording. I am not convinced I do have a solid grasp now on the security concern being discussed AFAIK 👍
ResponseThe below response was iterative as I worked through your response. You can probably just jump to the end as I think I identified a miscommunication and we may have been looking at the concern from different parts of the connection flow and trust. It can get a tad repetitive, no expectation to read and respond to everything here, I'm only being thorough without time to optimize 😅 Click to expand (Verbose)
And here I am thinking I had to write my own little TCP proxy program 😆 thanks for that example! 🥳
Correct 👍
This one is unrelated to PROXY protocol, Traefik is just routing layer 4 traffic from the client to the service. Normally knowing this is HTTP traffic you'd use the proper layer 7 router specifically for that and it'd handle this correctly. The service (Caddy) only trusts those headers when I have another setting configured to trust whatever headers are coming from Traefik. Realistically nobody should be deploying like this when trust matters (although I wouldn't be surprised if some have unintentionally). In the Client => Traefik => Caddy flow, when only routing traffic as layer 4 to Caddy:
Just to confirm, when you're expecting a specific protocol(s) it shouldn't be an issue right? For example I have noticed that if HTTP or TLS is being listened for, these fail if the client has provided an unexpected PROXY header. Whereas if you're optionally expecting PROXY header, or mandating one that should be processed then handling of HTTP or TLS which will go smoothly. When the malicious client provided their own PROXY header, and Traefik TCP forwarding additionally added it's own to provide Caddy, this resulted in two PROXY headers, so Caddy would fail from the malicious attempts to spoof the client IP. You'd know this better than I would though, a scenario where that's a valid concern, where a connection could initiate with data that would be mistaken as a PROXY header, but AFAIK if the receiver is handling a protocol where that would be invalid it's not going to be a problem? 🤷♂️ (so one would assume this context is more apparent as a conflict concern and prefer a
If the host is unconditionally trusted with it's connections to the receiver, then that host needs to ensure it does not allow clients to inject PROXY protocol header when connecting to it (assuming the TCP traffic routed would never be the kind to include legitimate data at the start of a connection that mirrors the header). Then there is no potential for spoofing?
You don't think it's too restrictive? If you follow That just requires a workaround via a separate port for the same purpose which is quite inconvenient. We already discuss reverse proxies which likewise allow routing connections via various rules but primarily supports a single port listener externally. Going further, provided the trusted host really can be trusted completely... it should be able to optionally provide the PROXY header. It's already trusted at this point, so the lack of it providing a header to the receiver shouldn't matter. Your concern is with the "dumb TCP forwarder" that is completely unaware of the PROXY protocol in traffic, which would make that pointless to consider as a trusted host? The receiver (Caddy):
Basically what I covered already above. This reads to me as it's ok to conditionally handle via trusted IPs. Yet not ok for a trusted IP to optionally omit the PROXY header when relevant (proxy to proxy). In the above Caddy listens on port 443 for HTTPS traffic.
Right... that's a policy though?
So perhaps the
I'm curious if there is a misunderstanding between us here too..?:
Given the context of Traefik being configured to disregard any PROXY header from a client connection, but otherwise continue the connection, we're good here? Your concern is primarily with Traefik blindly routing TCP connections where the client could add the PROXY header and the receiver Caddy would blindly trust that from Traefik. With that out of the way, the trust policy may be flexible as it's only concerned about WHEN to trust a PROXY header, but otherwise allow connections through without it when that is acceptable instead of rejecting them outright (which is what the spec currently demands with it's wording and you see in software like Postfix and Dovecot which requires separate ports as a result). If a host is trusted for connections with PROXY protocol, that is all that matters and it comes down to the policy which we can now agree is fine to be flexible.
Otherwise, I really am confused on the concern with the PROXY header being optional for Caddy and how enforcing the expectation of the PROXY header on Caddy's end resolves anything with connections from Traefik as a trusted host. It only makes sense to me that you've instead been focused on changes to Traefik, where the correct resolution is to not trust any clients connecting with PROXY header by default. |
“I apologize for such a long letter - I didn't have time to write a short one.” |
Specific set-ups might not necessarily be affected, because they perform validation. But a protocol specification cannot deal with specific set-ups, but must deal with the general case. PROXY protocol is mostly useful for dumb TCP forwarders, as when you're dealing with HTTP you have the much more powerful And those dumb TCP forwarders which is the prime use case for PROXY protocol are exactly those that are affected by optionally accepting a PROXY header, as I've outlined before. For your lengthy further reply, I'm cherry-picking specific points I consider important to clarify.
Yes. And if Traefik is adding its own PROXY header all the time, Caddy could just strictly require the existence of the PROXY header for Traefik's IP address, instead of trying to guess, thus making a misconfiguration much more evident.
Yesn't. PROXYv2 was specifically designed not to be confused with existing well-known protocols and thus trigger safe behavior. But that doesn't mean that there is no protocol where a PROXY header wouldn't be valid application data.
A very simple example would be a headerless and encrypted protocol using a pre-shared key. Such a protocol would be indistinguishable from random noise. A packet might encrypt to a valid PROXY header just by chance. I believe Wireguard would be such a protocol, but don't quote me on that. Another example would be a protocol consisting of an TLV encoding.
No, the gateway needs to ensure that it adds its own PROXY header, if the backend would accept the PROXY header from it. And if the backend accepts it, it might as well require it (as the protocol specification says).
A dumb TCP forwarder is be a trusted host if it reliably adds a PROXY header by itself. And again, if it does so, the backend can require the PROXY header to be present.
You read that correctly.
Dropping bytes that the client sent breaks protocols and could introduce security vulnerabilities, because the downstream backend might interpret the clients data differently, e.g. because a relevant TLV field is dropped. |
|
I've collapsed my original response as I don't want to take up your time any further 😅 Conclusion:
I'd still like to see the specification not enforce restrictions when they're not necessary, but with the spec clarification for a receiver, I can't think of any convincing real world scenarios right now 👍 I'll close the issue as I consider this resolved. Thanks for taking the time to clear everything up :) Full response
Understood, I was just establishing that the Random data mistaken as PROXY headerRedundant
I am only familiar with it being relevant when initiating a TCP connection, so I'd think that is not a concern for an established protocol? I'm sure there is probably scenarios where this sort of thing could happen by accident, but I'm only familiar with the identified scenario of blindly forwarding TCP traffic.
Fair, but in that case you could still attribute this context as part of a trust policy? In the sense that because of that potential mishap out of your own control, you must have strict enforcement of adding a PROXY header from the trusted host and the receiver enforcing Although, if the specification allowed for signing the PROXY protocol header, any compatible receiver would not run into this problem if verifying the signed bytes. Random data is not going to be mistaken then (the chance of that is ridiculous). Context matters - Flexibility can still be secure
Redundant
"Untrusted" hosts within a trusted internal network which do not need to connect via the external gateway can make direct connections to the service, but the specification doesn't permit the port to allow these connections to be exempt from providing the PROXY header.
While "might as well require it" does make sense and I tend to agree, there may be software like the Traefik example where:
Receiver service + sender trust managementMostly touching on some common concerns and clarifying the scope of trust for a receiver, which the wording in the current spec may be misinterpreted or affect implementation decisions, adding friction without a more flexible trust policy. Related: Response on the definition of a receiver and it's trust as per specEDIT: I was too focused on the public vs private here to even think about foreign connections being direct preserving the client IP if no proxy / hops are in between. Disregard that.
👍 From the PROXY protocol spec:
Ok so a receiver is not just the service + port, but additionally bounded by the IP of the sender (host connecting directly to the service):
If trusted hosts were policy based beyond Security via context / policyThe specification concern is that there are scenarios where more flexibility via a trust policy and other context presents no security issues, however the specification enforces expectations rather than deferring to policy. It appears that it is more appropriate as best practice advice to avoid any unexpected surprises?
Related: Response to dropping bytes / TLV field
This still seems to be heavily context dependent? I can see why it is safer to enforce requiring PROXY protocol headers from a trusted host, but I don't see why the specification needs insist it be mandatory? Dropping bytes:
I am not that familiar with TLV fields, but if they're part of the PROXY protocol header that is discarded from untrusted hosts, how does this introduce a security vulnerability or affect the backend? Why would a relevant TLV field exist in data we don't trust or expect? I can understand from the perspective that under a different context than I described, this could be a valid concern. I just don't see how it applies to the PROXY protocol specification for everyone. Rather, more of a best practice that you Take the advice for modulus keylength, some already advise >2048 bits or forecast to advise that going forward. 112-bits of symmetric security is quite a lot, the computation cost to attack that for a target like myself would be too expensive already. Likewise with entropy for passwords. I know what a safe lower bound is there, and I've seen others insisting 128-bit symmetric is not enough, wanting to use 4096-bit RSA (sometimes thinking it's only twice the strength of a RSA 2048-bit key). This is relevant to policy decisions, which is where I think the PROXY protocol specification could treat it as such. |
Some implementations like
go-proxyprotohave implemented a policy system for making it possible to accept connections without PROXY protocol, while only accepting the header from trusted hosts. This is what Traefik uses and Caddy is changing to in their 2.8 release.While documenting PROXY protocol support on a project, one of the reviewers referenced the PROXY protocol specification with the
MUSTwording that forbids this approach:haproxy/doc/proxy-protocol.txt
Lines 176 to 182 in 2ed53ae
While the
SHOULDencourages minimizing risk by restricting what hosts are trusted.It would seem that if you manage what hosts are trustworthy, that's what matters here when accepting PROXY headers, but those same hosts could just as well not provide the PROXY header and not have PROXY protocol apply?
Isn't the only concern really that the receiver must not use the PROXY header when the connection is from an untrusted host? Perhaps the specification should be revised to be more flexible on that point?
References:
go-proxyprotolanded a policy feature roughly a year ago related to internal vs external traffic to the same port motivated by kubernetes deployments.The text was updated successfully, but these errors were encountered: