How to reverse proxy websockets?

[tessus]

I am starting this discussion, because there are a few points that are not entirely clear to me. And maybe to others.

I am using the current main branch (at commit 44e9e1a). This code already has the support for websockets on the same port via rocket.
However, I am still using WEBSOCKET_ENABLED=true with the following Apache rewrite rules:

RewriteEngine On
RewriteCond %{HTTP:Upgrade} =websocket [NC]
RewriteRule /notifications/hub(.*) ws://localhost:3012/$1 [P,L]
ProxyPass / http://localhost:8888/

In a PR discussion my question came up how to use the new implementation with WEBSOCKET_ENABLED=false.

While the rewrite rule to port 3012 might not be necessary anymore, the protocol still needs to be changed, doesn't it?

The ProxyPass / http://localhost:8888/ would pass traffic to http://localhost:8888/:

https://proxy.example.com/notifications/hub?data...  -> http://localhost:8888/notifications/hub?data...

But web sockets use a different protocol. So what would happen, if a websocket connection was routed to http://....? Shouldn't it be proxied to ws://....?

Thoughts?

[BlackDex]

Looking at the official documentation, the rewrite rule needs to be changed a bit. https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html

ProxyPass / http://localhost:8888/
RewriteEngine on
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule ^/?(.*) "ws://localhost:8888/$1" [P,L]

There are other reverse proxies which handle this then selfs, or need some other small adjustments.

[BlackDex]

btw, i actually think this would have been better from the start.

[tessus]

Thanks, although there is a slight error in the last line. The pattern is missing. e.g. ^/(.*) could work.

Just as a fun fact, the P flag implies that it is the last rule, so one doesn't need the L flag. But it doesn't hurt either.

[tessus]

When I get to it, I will run a few tests. No ETA though. I usually either use HAproxy or Apache httpd as a reverse proxy. Traefik I only use in Nomad clusters.

This discussiom should also motivate people to update the proxy examples wiki page. ;-)

Unless somebody beats me to it, I'll update the Apache proxy cfgs (by adding this option and not deleting the previous one) after my tests.

[BlackDex]

In the end, i would like to have a clean overview of examples.
Maybe we should move older examples to a lower section on the page. Else it will get very messy and people do not know which version they need to use in the end.

[stefan0xC]

While the rewrite rule to port 3012 might not be necessary anymore, the protocol still needs to be changed, doesn't it?

Nothing should be running on port 3012 with WEBSOCKET_ENABLED=false. So the the important part would be to replace the port 3012 with port 8888 in your example (i.e. sending everything to the rocket instance). As far as I understand it the protocol still needs to be changed if you are using mod_proxy_wstunnel.

I have not tested this yet but since the websocket URL (where the upgrade happens) is known and if your using Apache HTTP Server 2.4.47 (or newer) you might be able to simplify the configuration further using something like the following mod_proxy directive:

ProxyPass /notifications http://localhost:8888/notifications upgrade=websocket
ProxyPass / http://localhost:8888/

[stefan0xC]

Ah, I have not seen that. Can't we assume that every endpoint under the /notifications path might upgrade to a websocket connection?

[BlackDex]

Why should we? I would rather have a more general / standard config then a fully customized for Vaultwarden config.

[stefan0xC]

Because it seems like a safe assumption given that there is currently only one path (and there will probably only be two paths) to which this applies? I think that the assumption that every path might upgrade to a websocket connection is a bit overkill (even if the overhead of checking for those headers is probably negligible).

[BlackDex]

The problem is, if there are new ways or endpoints add, people need to change there reverse proxy config again. It's better to have a general working example.

[tessus]

stefan0xC was already on the right track.

As of Apache HTTP Server 2.4.47, the following is all that is needed to upgrade websocket connections. No more rewrite condition and rules.

ProxyPass / http://localhost:8888/ upgrade=websocket

There is no need to specify a /notification path. Upgrade only takes place when requested
AND accepted/switched by the backend server.

I'll add another entry below for Apache. With time maybe people add the setups for other reverse proxies as well. In which case at one point we just have to copy and paste that info to the wiki. ;-)

[ShlomiD83]

So what is the recommended way of using websockets?
Remove the env variable, the port mapping and the Nginx proxy manager config for /notifications/hub?

[tessus]

Apache httpd

In vaultwarden set WEBSOCKET_ENABLED=false

Apache HTTP Server >= 2.4.47

ProxyPass / http://localhost:8888/ upgrade=websocket

Apache HTTP Server < 2.4.47

ProxyPass / http://localhost:8888/
RewriteEngine   on
RewriteCond     %{HTTP:Upgrade}       websocket                   [NC]
RewriteCond     %{HTTP:Connection}    upgrade                     [NC]
RewriteRule     ^/?(.*)               "ws://localhost:8888/$1"    [P,L]

[netboy3]

How would those rules work with a non-root domain setup, I.E. https://xxx.yyy.zzz/bitwarden/....
My environment is set with DOMAIN=https://xxx.yyy.zzz/bitwarden, but vaultwarden seems to barf with 404's on any requests coming for ".../anonymous-hub/?"

[BlackDex]

Vaultwarden does have a setting for the sub-path, that needs to be done in the DOMAIN config, as described on the wiki.

[netboy3]

because vaultwarden does not have a config option to specify a subpath

But it does. That's what the DOMAIN variable does. It prefixes the whole Rocket namespace with the subpath specified there.

[tessus]

This is exactly my point. The DOMAIN variable is the external request URI. vaultwarden does not listen on the external URI. The reverse proxy does. vaultarden listens on ROCKET_ADDRESS and ROCKET_PORT. There is no variable for ROCKET_PATH.

Unless the vaultwarden server uses the external URI to build a local URI for the app server. Which is not how that usually works.

Look at any other server that can be reverse-proxied. There are usually 2 variables: one for the external request URI and one for the local app server path. e.g. they could be different.

[BlackDex]

Again, we use the DOMAIN variable. It needs to match. We extract the path from that variable and prepend that to the endpoints.

[netboy3]

Here's how it's done in the code:
First, the path is extracted from the DOMAIN URL and assigned to domain_path (settings struct member):

vaultwarden/src/config.rs

Line 428 in bc26bfa

domain_path: String, false, auto, |c| extract_url_path(&c.domain);

Then domain_path is being assigned to a basepath local variable:

vaultwarden/src/main.rs

Line 514 in bc26bfa

let basepath = &CONFIG.domain_path();

Which in turn is used as a prefix when mounting/registering routes/catchers during Rocket launch:

vaultwarden/src/main.rs

Lines 527 to 536 in bc26bfa

	.mount([basepath, "/"].concat(), api::web_routes())
	.mount([basepath, "/api"].concat(), api::core_routes())
	.mount([basepath, "/admin"].concat(), api::admin_routes())
	.mount([basepath, "/events"].concat(), api::core_events_routes())
	.mount([basepath, "/identity"].concat(), api::identity_routes())
	.mount([basepath, "/icons"].concat(), api::icons_routes())
	.mount([basepath, "/notifications"].concat(), api::notifications_routes())
	.register([basepath, "/"].concat(), api::web_catchers())
	.register([basepath, "/api"].concat(), api::core_catchers())
	.register([basepath, "/admin"].concat(), api::admin_catchers())