rpxy behind rpxy-l4

[tribals]

I'm trying to set up rpxy as "all-the-HTTP" TLS terminating server. Unfortunately, I must multiplex protocols on port 443, so I decided to use rpxy-l4 to do that. On the other hand I need to keep original client IPs for rpxy, so I decided to proxy Web traffic from rpxy-l4 using HAProxy PROXY protocol. To do so, I've enabled it in prxy's & l4 configuration:

[experimental.tcp_recv_proxy_protocol]
trusted_proxies = ["127.0.0.1/32", "::1/128"]

[protocols."https"]
protocol = "tls"
target = ["localhost:2443"]
alpn = ["http/1.1", "acme-tls/1"]
tcp_send_proxy_protocol = "v2"

On the third hand, I need to redirect all the plain HTTP to HTTPS, so to do that I need to have some server listening on TCP/80.

I'm wonder how could I implement that with rpxy-l4/rpxy tandem?

The first idea was to make rpxy-l4 to listen on TCP/443 (HTTPS) and UDP/443 (HTTP/3) and proxy to rpxy using PROXY protocol, and make rpxy to terminate TLS, renew ACME certs, and listen on TCP/80 for plain HTTP (on it's own) (with the sole purpose of redirecting to HTTPS).

Unfortunately, this doesn't make it. Three problems there.

1. Enabling PROXY protocol makes rpxy to expect it on plain HTTP TCP/80 as well, which obviously not the case for external request.

2. I need rpxy to listen for TLS/QUIC only on localhost, as everything will be proxied to it, but can't achieve that with current configuration:

listen_port = 80
listen_port_tls = 2443
listen_address_v4 = ['127.0.0.1', 'a.b.c.d']  # a.b.c.d is real IP here
listen_address_v6 = ['::1', 'cafe:babe::1']  # as well IPv6 here

This make rpxy to listen on localhost and public IP as well, while it needs to listen on public IPs only for plain HTTP on TCP/80. This compromises security as it enlarges attack surface. (Still I can ensure security through firewall, but...)

3. In theory, I can make rpxy-l4 to listen on TCP/80 and proxy to rpxy the same way, as it has support for HTTP protocol (instead of TLS), but it can only listen on single port for now:

# TODO: Is it nice to have multiple listen ports?
listen_port = 443

I'm wonder if this is ever achievable? What do you think?

Some options I've thought about:

1. Ignore HTTP (TCP/80) completely as it seems most browsers already shipped with "strict HTTP security" policy, always redirecting to HTTPS.

The problem that I'm thinking plain HTTP is required for ACME to work. Is that correct? I'm not much familiar with TLS-ALPN-01 challenge.

2. Make rpxy-l4 to listen & proxy on TCP/80 somehow.

Any considerations are appreciated.

😊

[junkurihara]

Thanks for the detailed writeup. You've already mapped the problem space accurately, down to reading the listen_port TODO in rpxy-l4, so let me address each point directly.

First the ACME concern, because I think it dissolves most of the tension: rpxy does not need plain HTTP on :80 for certificate issuance or renewal. It uses TLS-ALPN-01 (RFC 8737), not HTTP-01, which is exactly what the acme-tls/1 entry in your rpxy-l4 alpn list is for. The challenge completes entirely over :443: the ACME CA opens a TLS connection negotiating the acme-tls/1 ALPN, and rpxy answers with a self-signed certificate carrying the key-authorization digest. In your topology this works as-is. rpxy-l4 sees the acme-tls/1 ClientHello, matches your [protocols."https"] rule, and forwards it (PROXY-protocol-wrapped) to rpxy, which terminates it. So you can drop :80 from rpxy with no impact on certs. The only remaining reason to keep :80 is the HTTP→HTTPS redirect, and as you note, HSTS / HTTPS-first behavior in current browsers has made that largely cosmetic.

On the two rpxy limitations you hit (PROXY-protocol expectation and the shared listen address): I did look at per-listener configuration for these earlier and decided against it, and I'm not planning to add it for now, though if the internal architecture is ever reworked, it's something I'd revisit at that point. The reasoning behind that decision: tcp_recv_proxy_protocol / trusted_proxies is not really a property of an individual socket; it is a declaration that "this instance sits behind a trusted L4 frontend." A per-listener form would permit a config where a publicly-exposed socket trusts PROXY headers, which is a source-IP-spoofing footgun, the trusted_proxies allowlist exists precisely to bound that blast radius, so loosening it per-socket would cut against that. The shared listen address follows from the same deployment posture. So rather than relaxing these, the cleaner path is to keep all ingress to rpxy uniform: everything arrives via rpxy-l4, PROXY-protocol-wrapped, and rpxy listens only on localhost.

That leaves the redirect, and my recommendation is one of two shapes:

1. Drop :80 entirely and rely on HSTS. Simplest, and with TLS-ALPN-01 there is no functional loss.

2. If you want the redirect, run a second, dedicated rpxy-l4 instance on :80 that forwards (PROXY-protocol-wrapped) to rpxy's plain-HTTP listener on localhost. rpxy still performs the redirect, but now receives :80 traffic with a PROXY header just like :443, which incidentally makes your original problem H3 #1 disappear, because there are no longer any direct :80 connections to rpxy that lack the header.

On why I'd prefer two instances over a multi-port rpxy-l4: for TCP the difference is mostly config-table bookkeeping, since each accepted connection is an independent spliced pair. The real cost is on the UDP/QUIC side. UDP has no connection, so rpxy-l4 keeps its own flow table keyed on (client src, listen socket) -> upstream socket in order to demux return datagrams back to the correct client and the correct listener. Adding multiple listen ports multiplies that state along the listen-socket dimension. Socket lifecycle, timeouts, cleanup, and return-path demux all grow with it, which is a meaningful increase in live state and bug surface inside a single process. Splitting into separate instances collapses that dimension to a constant per process and lets OS process isolation handle the separation for free. Since rpxy-l4 is purely L4 and shares no cross-port state (no TLS termination, no cert cache — those live in rpxy), a second instance costs only one more config file and one more process; a systemd template unit makes the operational overhead negligible. That trade is clearly favorable for a 443+80 setup. Multi-port would only begin to pay off with many listen ports, which is not rpxy-l4's target shape.

In short: keep rpxy on localhost behind rpxy-l4 with PROXY protocol, let TLS-ALPN-01 handle ACME on :443, and either drop :80 or give it its own rpxy-l4 instance.

[tribals]

I've already tested this setup (according to my initial idea), and it works fine. So, probably I will give up on listening to TCP/80.

Another issue arose that I must specify https_redirection_port for Alt-Svc to work properly, but can't do that if I don't specify listen_port:

ERROR rpxy service exited: Invalid configuration: https_redirection_port can be explicitly specified only when both http_port and https_port are specified

So, for now I'v specified "dummy" listen_port only to have proper Alt-Svc (offering HTTP/3).

And, when reaching upstream for clients coming through HTTP/3, I'm not able to make it see real client IPs - upstream reports they're always coming from localhost. I'm wonder does rpxy modify HTTP headers for QUIC <-> HTTP upstreams?

Also, it is not clear for me is it worth to bother with HTTP/3 at all? Old HTTP and HTTP/3 have slightly different semantics ("streams" etc), so I'm not sure is there any gains for clients to go through HTTP/3 if in the end it will reach upstream through "good old" HTTP?

[junkurihara]

Thanks! this is a good catch.

I'll fix this in 0.13 as part of a breaking config cleanup. Alt-Svc should not require a dummy plaintext listen_port; it should be advertised on secure responses for non-mTLS apps when H3 is enabled.

The advertised H3 port will default to listen_port_tls, so no extra setting is needed when clients see the same port rpxy listens on. An override will only be needed when rpxy is behind something like rpxy-l4 / LB / NAT / port mapping and the client-visible HTTPS/H3 port differs from listen_port_tls.

mTLS apps will continue not to advertise H3.

[junkurihara]

Solved by #594 !

[tribals]

Yaaay!