Problems with Websocket after new Update

[xKugeki]

Hey everyone :)

I just upgraded my Vaultwarden to 1.29.0 and I removed WEBSOCKET_ENABLED=true from my Env.
As I understood: We dont have to set this in our Env because it will work now over Port 80

Releasenotes:
WebSocket notifications now work via the default HTTP port. No need for WEBSOCKET_ENABLED and a separate port anymore. The proxy examples still need to be updated for this. Support for the old websockets port 3012 will remain for the time being.

But for some reason my Desktop App/Browser App isn't syncing with the Web-Vault (just after reloading).
I allready put the WEBSOCKET_ENABLED=true back in my Env.. but same problem appears^^

Im using Apache2 Reverse with the following sites-enabled config: (I removed IPs and Domains ^^)
DOMAIN.com.conf

<VirtualHost *:80>
        ServerName DOMAIN
        ServerAlias DOMAIN

#        ProxyPass / http://LOCAL_IP:8083/
#        ProxyPassReverse / http://LOCAL_IP:8083/

        RewriteEngine On
        RewriteCond %{HTTPS} off
        RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
        RewriteCond %{SERVER_NAME} =DOMAIN
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<VirtualHost *:443>
        ServerName DOMAIN
        ServerAlias DOMAIN

    # These settings are definitely needed
    SSLEngine On
    SSLProxyEngine On
    ProxyRequests Off
    SSLProxyVerify none
    SSLProxyCheckPeerCN off
    SSLProxyCheckPeerName off
    SSLProxyCheckPeerExpire off


        SSLEngine on
        SSLProxyEngine on

        ProxyPass / https://LOCAL_IP/
        ProxyPassReverse / https://LOCAL_IP/

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        SSLCertificateFile /etc/letsencrypt/live/DOMAIN/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/DOMAIN/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

I hope someone can help me :)

~Have a nice day! :)

[BlackDex]

I suggest to read #3598 (comment)

[xKugeki]

I tried everything there :/
I changed to:
ProxyPass / http://LOCAL_IP:8083/ upgrade=websocket
I also tried:
ProxyPass / http://LOCAL_IP:80/ upgrade=websocket
I also tried:
ProxyPass / http://LOCAL_IP:3012/ upgrade=websocket

and I tried meanwhile both:

WEBSOCKET_ENABLED=true

WEBSOCKET_ENABLED=false

I put the lines under <VirtualHost *:80> is that correct?
I got Apache Version 2.4.55 (Ubuntu)

[xKugeki]

how exactly do i set this up for Port 80 usage?
Should the port remain on 3012?
Cant change it to 80 bc it is in use. (the error message i get^^)

[xKugeki]

@BlackDex when I login i can see the the error: 502 Bad Gateway with the network inspect

[BlackDex]

@xKugeki under your VirtualHost *:80 will not work, since that will redirect to HTTPS.

You need to put it under your *:443 config as that is the part that actually passes the calls through to Vaultwarden.

<VirtualHost *:80>
        ServerName DOMAIN
        ServerAlias DOMAIN

        RewriteEngine On
        RewriteCond %{HTTPS} off
        RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
        RewriteCond %{SERVER_NAME} =DOMAIN
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<VirtualHost *:443>
        ServerName DOMAIN
        ServerAlias DOMAIN

        # These settings are definitely needed
        SSLEngine On
        SSLProxyEngine On
        ProxyRequests Off
        SSLProxyVerify none
        SSLProxyCheckPeerCN off
        SSLProxyCheckPeerName off
        SSLProxyCheckPeerExpire off


        SSLEngine on
        SSLProxyEngine on

        ProxyPass / https://LOCAL_IP/ upgrade=websocket
        ProxyPassReverse / https://LOCAL_IP/

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        SSLCertificateFile /etc/letsencrypt/live/DOMAIN/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/DOMAIN/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

[xKugeki]

Hey I did so and restartet the apache2 service.
But i still get the "502 Bad Gateway" Error :/

[BlackDex]

That means your ProxyPass is wrong.

I suggest to read the examples here: https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples

[xKugeki]

@BlackDex thank you :)
I will give it a try and try to use the example

[tessus]

@xKugeki

• what is your ROCKET_PORT in the vaultwarden config
• what is your ROCKET_ADDRESS in the vaultwarden config
• is Apache running on the same machine as vaultwarden

You only have to set the following in your Apache virtual host for *:443:

ProxyPass / http://IP_OF_VW_SERVER:ROCKET_PORT/   upgrade=websocket

If Apache is running on the same machine as vaultwarden, you can just use localhost or 127.0.0.1 for IP_OF_VW_SERVER.

[tessus]

You are running in Docker. Docker uses an overlay network where each Docker container has its own IP address, in this case 172.19.0.0/16. I don't know, you have to check your Docker daemon configuration.

However, your setup is wrong. You still have websockets active on port 3012. You also use a mapping. The container listens on port 8083 while the service in the container is running on port 80. This is perfectly fine, but this is the reason why your reverse proxy config is wrong.
Also, rocket TLS is not active (which is good), but this also means you cannot use a TLS connection to the backend service.

Do this:

version: '3'

services:
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    ports:
      - 8083:80
    environment:
      - WEBSOCKET_ENABLED=false # not needed anymore
      - ADMIN_TOKEN=$$argon2XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
      - SIGNUPS_ALLOWED=false
      - DOMAIN=https://DOMAIN.COM
      - DATABASE_URL=mysql://LINK
      - ORG_GROUPS_ENABLED=true
      - WEB_VAULT_ENABLED=true
      - ROCKET_PORT=80
    volumes:
      - ./vw-data:/data

Apache config:

<VirtualHost *:80>
        ServerName DOMAIN_NAME
        ServerAlias DOMAIN_NAME

        RewriteEngine On
        RewriteCond %{HTTPS} off
        RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
        RewriteCond %{SERVER_NAME} =DOMAIN_NAME
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

<VirtualHost *:443>
        ServerName DOMAIN_NAME
        ServerAlias DOMAIN_NAME

        SSLEngine On

        ProxyPreserveHost On
        ProxyRequests Off
        RequestHeader set X-Real-IP %{REMOTE_ADDR}s

        ProxyPass / http://LOCAL_IP:8083/   upgrade=websocket

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
        SSLCertificateFile /etc/letsencrypt/live/DOMAIN_NAME/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/DOMAIN_NAME/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

P.S.: I have a few appointments this afternoon, so I won't be able to reply the next 4 hours or so.

[xKugeki]

Hey @tessus thanks for your help :)
I edited my docker-compose.yml and my apache conf
but as soon as i reload or restart the apache2 service i cant enter my site anymore.

just changing the ProxyPass back to https://LOCAL_IP/ upgrade=websocket will allow me to enter my site.

But I can enter my Vault via http://LOCAL_IP:8083 in the webbrowser

[tessus]

I think I have to stop here, because this makes no sense.

what is running on port 443 on your VM that runs the vaultwarden container? It can't be vaultwarden, because it is running on port 8083 according to the docker compose file.

I think you might be trying to reverse proxy something else. What do you mean by "my site"?

We were trying to reverse proxy just vaultwarden under a subdomain, weren't we?
You see, you might be mixing up a lot of different things. If you are trying to use the same virtual host for 2 different backend services, then you have to use a sub location for either one of them.

You are omitting so much information by retracting domain name and IP addresses that nothing makes sense anymore. I understand that you want to do that, but in this case, use fake hostnames and properly named IP substitutions. eg: vault.example.com, IP-OF-VW-VM, and so on.

I think we are still missing a lot of information about what you actually want to do. Additionally you keep contradicting yourself.

Also, you said you upgraded vaultwarden to 1.29.0, so your previous setup still works. There is no reason to change a working setup. And we are obviously not getting anywhere, especially since the change from websockets on 3012 to websockets on the same port as rocket is

• a removal of 3 lines in the apache reverse proxy config, and changing 1 line
• setting 1 config var for vaultwarden to false

That is all there is to it. So I am more than a bit confused as to why this is not working. I am seriously at a loss.

I don't want to offend you, but I believe this can only be solved, if someone sits down next to you and gives you an overview of networking, docker, reverse proxying, and then does the changes in your setup. I really doubt that we will be able to solve this via this discussion. We have already given you all the right answers.
If they do not work, you are trying to do something else or the information you are giving us is incorrect.
I just don't know where to go from here.

[xKugeki]

Hey @tessus Im sorry for that, I really appreciate your help :)
Unluckily I didnt setup the Proxy Server, a friend did that and he is in vacation at the moment.
So i will wait till he gets back and rebuild that apache2 stuff ^^

Everything works so far, just Websocket wont work.
I got my personal own setup with NPM and Vaultwarden - actually no problems with websocket ^^
Everything works fine there. But my knowledge with apache is so damn limited :/

But I want to get the setup working with apache2 for my friend ^^
I will rebuild that and update here if that worked :)

Thank you very much for your help and sorry for the circumstances

I wish you a great day!

P.S: my site = vaultwarden ^^

[tessus]

No worries. As I mentioned before, I don't know where to go from here, otherwise I'd help you to figure this out.
You are running the vaultwarden docker container on port 8083, yet somehow - in a very mysterious way - you actually connect to it via port 443. This makes no sense, thus I can't help with this.

[BlackDex]

If you are not getting it to work, i suggest to revert to your old setup. That should still work.

[xKugeki]

@tessus
I checked the machine vaultwarden is running on and found that my friend configured something in /etc/nginx/conf.d/vaultwarden.conf

domain.com = our domain, just swapped with "domain.com"
the rest is the original conf

server {
  listen 443 ssl;
  server_name domain.com;

  ssl_certificate /home/VW-Certificate/fullchain.pem;
  ssl_certificate_key /home/VW-Certificate/privkey.pem;

  # Specify SSL config if using a shared one.
  #include conf.d/ssl/ssl.conf;

  # Allow large attachments
  client_max_body_size 128M;

  location / {
    proxy_pass http://127.0.0.1:8083;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  location /notifications/hub {
    proxy_pass http://127.0.0.1:3012;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
   }

  location /notifications/hub/negotiate {
    proxy_pass http://127.0.0.1:8083;
  }

}

I actually dont know where that comes from :/
Because we use a Proxy Server (Apache2) on another machine.

I think that causes one of the problems.
But I dont know if i can just delete that or if that is correct ^^'

[BlackDex]

As mentioned before, read the Proxy Examples Wiki page: https://github.com/dani-garcia/vaultwarden/wiki/Proxy-examples

There you will see everything you need to know.
But to be clear, nothing has changed in Vaultwarden which could have caused an issue, since the old way is still working as before.
The latest version of Vaultwarden just has the feature to remove the old 3012 port dedicated for WebSocket's and it can now be run on the default port, same as the rest of the whole server.

[xKugeki]

Yeah I know that this isnt because of the the update ^^
I mean everything works right now expect websocket but that didnt worked before either on that machine ^^

Im just wondering right now if the setup this way:
UbuntuMachine1 with Docker and Vaultwarden with Port 8083 installed + Nginx installed with the Config above ist correct?

Because on UbuntuMachine2 is the Apache2 Reverse Proxy. I think there is one to much.

In my own vaultwarden i used nginx proxy manager on the same machine and that was easy to setup.
But how does it change when your reverse proxy is on another machine ?

[tessus]

Hmm, ok, it seems you are reverse proxying twice. Not really an issue, but if you do stuff like that, you should know what you are doing. I think this is out of this discussion's scope, but I can certainly help you with this when I have more time.

In reality there are many reasons, why there are multiple reverse-proxy hops. I have seen as many as 3 and 4. But in this case, I doubt it gives you any advantage nor does it make any logical sense.
You are not load balancing on a massive scale, by which I mean like more than a 1,000 nodes.

[xKugeki]

Okay after rechecking the config and using the examples given from you It seems like the problem was caused by the nginx config ^^

Adding

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      "";

and changing

  location / {
    proxy_pass http://127.0.0.1:8083;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }

  location /notifications/hub {
    proxy_pass http://127.0.0.1:3012;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
   }

  location /notifications/hub/negotiate {
    proxy_pass http://127.0.0.1:8083;
  }

}

to:

location / {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";


    proxy_pass http://LOCAL_IP:8083;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
  }
#
#  location /notifications/hub {
#    proxy_pass http://LOCAL_IP:3012;
#    proxy_set_header Upgrade $http_upgrade;
#    proxy_set_header Connection "upgrade";
#   }
#
#  location /notifications/hub/negotiate {
#    proxy_pass http://LOCAL_IP:8083;
#  }

fixed the problem for me. Had some problems after that with the live sync in organisations but that was because of #3413 so that´ll have to wait for first :)

Thanks again to you @BlackDex @tessus and once again have a nice day :)