Router: Websocket support over a TLS terminating reverse proxy

[marvkis]

Hi,

I've been working on helm charts openziti/helm-charts#234 to get a 'kubernetes browzer' support. Since there is (imho) no need for mTLS on the WSS endpoint, it should work through a reverse proxy doing the TLS termination.
This would make things easier in the kubernetes world, as the certificate renewal and handling works very well with the existing ingress controllers / reverse proxies. So I tried to set it up like that way for the WSS endpoint, but the communication between the NGINX reverse proxy and the WSS endpoint failed.

On the router I see this logged when I try to access the WSS endpoint:

[74385.266]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3023]: {remote=[10.42.0.186:38284] error=[not handler for requested protocols ]]} handshake failed
[74385.266]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3023]: {remote=[10.42.0.186:38300] error=[not handler for requested protocols ]]} handshake failed
[74385.267]   ERROR transport/v2/tls.(*sharedListener).processConn [tls:0.0.0.0:3023]: {remote=[10.42.0.186:38302] error=[not handler for requested protocols ]]} handshake failed

These are the logs on the NGINX side:

2024/07/02 20:39:45 [error] 1384#1384: *29019299 SSL_do_handshake() failed (SSL: error:0A000438:SSL routines::tlsv1 alert internal error:SSL alert number 80) while SSL handshaking to upstream, client: 127.0.0.1, server: wss.external.freaks.de, request: "GET / HTTP/2.0", upstream: "https://10.42.0.178:3023/", host: "wss.external.freaks.de"
2024/07/02 20:39:45 [error] 1384#1384: *29019299 SSL_do_handshake() failed (SSL: error:0A000438:SSL routines::tlsv1 alert internal error:SSL alert number 80) while SSL handshaking to upstream, client: 127.0.0.1, server: wss.external.freaks.de, request: "GET / HTTP/2.0", upstream: "https://10.42.0.178:3023/", host: "wss.external.freaks.de"
2024/07/02 20:39:45 [error] 1384#1384: *29019299 SSL_do_handshake() failed (SSL: error:0A000438:SSL routines::tlsv1 alert internal error:SSL alert number 80) while SSL handshaking to upstream, client: 127.0.0.1, server: wss.external.freaks.de, request: "GET / HTTP/2.0", upstream: "https://10.42.0.178:3023/", host: "wss.external.freaks.de"
127.0.0.1 - - [02/Jul/2024:20:39:45 +0000] "GET / HTTP/2.0" 502 150 "-" "curl/8.6.0" 36 0.003 [openziti-core-router-listener-edge-wss-443] [] 10.42.0.178:3023, 10.42.0.178:3023, 10.42.0.178:3023 0, 0, 0 0.001, 0.000, 0.001502, 502, 502 3939e56ebe6d53a8734621aabba1919d

I started digging into this and playing around with openssl gave me an idea:

# doesn't work
openssl s_client -connect 10.42.0.246:3023
# works
openssl s_client -connect 10.42.0.246:3023  -alpn http/1.1

Looks like the router needs ALPN headers in the handshake. But it looks like NGINX doesn't have ALPN support on the backend side:
https://serverfault.com/questions/765258/use-http-2-0-between-nginx-reverse-proxy-and-backend-webserver

The error message we see in the logs is fired here: https://github.com/openziti/transport/blob/0666f1970ea9ab620fdffd5d902719941cb34c7d/tls/listener.go#L344

During my research I also came across traces indicating that there is a 'ws' variant, so I tried it with address: ws:0.0.0.0:3023 - but this only produces this error:

[49200.245]    INFO ziti/router/xgress_edge.(*listener).Listen: {address=[ws:0.0.0.0:3023]} starting channel listener
[49200.245]   FATAL ziti/router.(*Router).startXgressListeners: error listening [edge] (transport.ws not supported. use transport.wss)

Is there any way to make this work? Is the ALPN support a hard requirement or could it be optional? Or could the websocket port be made available over HTTP so that the termination is done at the reverse proxy and the internal traffic is unencrypted? (As I understand it, the webassembly creates an mTLS connection to the router using websocket as the transport layer).

Thanks & Bye,
Chris

[qrkourier]

As I understand the Ziti security model, edge clients must present a trusted client certificate to the router edge, WebSocket listener in this case, when they are creating a channel for a Ziti service they're authorized to dial or terminator for a service they're authorized to bind.

If that's accurate, then it's essential for the WebSocket listener to terminate TLS so that mTLS negotiation can succeed.

BrowZer (ZBR) clients obtain an ephemeral client certificate from the Ziti controller after "bootstrapping" with OpenID Connect. That's the cert they present to the WebSocket listener.

The controller's client API and the router's WebSocket listener must both present a publicly-trusted server certificate because the ZBR client is running inside a normal web browser that can not be configured to trust Ziti's root CA.

Only the router's WebSocket listener must terminate TLS, however, because ZBR clients never present the client certificate to the controller's client API.

EDIT: I stand corrected! Now I think the router's WebSocket TLS listener will work normally behind a reverse proxy that also provides server TLS. This is because no client cert is required to negotiate TLS with the WebSocket listener, only server TLS. The ZBR running inside the normal web browser will then negotiate mTLS for the edge transport protocol, presenting the ephemeral client certificate obtained from the Ziti controller, inside that WebSocket (server) TLS tunnel.

[qrkourier]

I'm working on proving this out in preparation for a development round on BrowZer deployments and docs.

[qrkourier]

The WebSocket TLS listener should complete the TLS handshake when the ClientHello does not specify an ALPN, as is typical for LB upstream/backend requests, especially when it's listening on a separate TCP port (not shared with other protocols, hence no possibility of ambiguity in protocol handler).

However, there's no estimate available for when the current behavior will change. For now, the server terminates the TLS handshake with an error when the client doesn't specify the ALPN protocol.

GitHub issue for the underlying cause: #2466

This problem does not impact the controller's client API; I've only encountered it with the router's WSS listener, which is required for BrowZer. The router deployments support terminating TLS with a trusted certificate with identity.alt_server_certs and a distinct advertised host for ZBR clients in their config YAML, which a TLS passthrough LB can use to route requests to the router's WSS port.

[qrkourier]

The consensus is that it's easier to explain (and document) a rubric like "Ziti requires TLS termination" than to clarify the nuances of when TLS termination is actually required.

I'm accepting/refining alt server cert support in the router and controller Helm charts, and this remains a feature of the other deployments for Linux and Docker, though it does require adjusting their configuration YAML to declare file paths to the publicly-trusted alternative certs.

TL;DR After #2466 is resolved, it will be possible (as an undocumented capability) to reverse-proxy the router's wss listener.

The documented guidance will be to mount the trusted certs directly on the router (container, kernel namespace, etc.) and use only TLS-passthrough load balancers to publish the listener.