[bug] ui: vault empty after login until refresh

[brandon1024]

Subject of the issue

Immediately after login in the UI, the vault appears to be completely empty. The vault entries populate correctly after a refresh and another login attempt.

I'm able to reproduce this consistently. Some errors appear in the console (see screenshot).

Deployment environment

• vaultwarden version: 1.29.1

• Install method: docker image (docker.io/vaultwarden/server:latest: bf9841431610)

• Clients used: web vault, Chrome Version 115.0.5790.170 (Official Build) (64-bit)

• Reverse proxy and version: nginx (docker.io/library/nginx:latest : 904b8cb13b93)

• MySQL/MariaDB or PostgreSQL version: N/A

Steps to reproduce

1. In a new tab, navigate to the login page for your deployment of Vaultwarden.
2. Enter your email address.
3. Enter your password.

Expected behaviour

1. Vault entries appear correctly in the UI.

Actual behaviour

1. No vault entries appear in the UI.
2. Errors appear in the developer tools console.

To fix:

1. Refresh the page.
2. Re-enter your password.
3. Vault entries appear.

Troubleshooting data

[BlackDex]

Your reverse proxy is replacing the Content Security Policy (CSP).
That is what is causing the issues.

Either something like ModSecurity or a WAF or something else adjusting them.

Vaultwarden already sends the correct headers, and modifying them will break the web-vault and probably other clients.

[brandon1024]

Thanks for the quick response!

I don't believe that's the case. The headers are the same regardless of whether I go through the proxy or not. My proxy configuration is pretty basic too (see below). I don't have any firewall like ModSecurity or WAF between the client and nginx. This also wouldn't explain why it works after a page refresh.

Here's a HEAD of the login page through the proxy:

HTTP/1.1 200 OK
Server: nginx/1.23.3
Date: Tue, 15 Aug 2023 12:19:17 GMT
Content-Length: 0
Connection: keep-alive
x-frame-options: SAMEORIGIN
permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
x-content-type-options: nosniff
referrer-policy: same-origin
x-xss-protection: 0
content-security-policy: default-src 'self'; base-uri 'self'; form-action 'self'; object-src 'self' blob:; script-src 'self' 'wasm-unsafe-eval'; style-src 'self' 'unsafe-inline'; child-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ; img-src 'self' data: https://haveibeenpwned.com https://www.gravatar.com ; connect-src 'self' https://api.pwnedpasswords.com https://api.2fa.directory https://app.simplelogin.io/api/ https://app.anonaddy.com/api/ https://api.fastmail.com/ https://api.forwardemail.net ;
cache-control: no-cache, no-store, max-age=0

Bypassing the proxy:

HTTP/1.1 200 OK
server: Rocket
x-frame-options: SAMEORIGIN
permissions-policy: accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), payment=(), picture-in-picture=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
x-content-type-options: nosniff
referrer-policy: same-origin
x-xss-protection: 0
content-security-policy: default-src 'self'; base-uri 'self'; form-action 'self'; object-src 'self' blob:; script-src 'self' 'wasm-unsafe-eval'; style-src 'self' 'unsafe-inline'; child-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-src 'self' https://*.duosecurity.com https://*.duofederal.com; frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ; img-src 'self' data: https://haveibeenpwned.com https://www.gravatar.com ; connect-src 'self' https://api.pwnedpasswords.com https://api.2fa.directory https://app.simplelogin.io/api/ https://app.anonaddy.com/api/ https://api.fastmail.com/ https://api.forwardemail.net ;
cache-control: no-cache, no-store, max-age=0
content-length: 0
date: Tue, 15 Aug 2023 12:18:49 GMT

Nginx Configuration:

events {}
http {
        server {
                listen 8080;
                listen [::]:8080;

                server_name vault.home.brandonrichardson.ca;

                listen 8443 ssl;

                ssl_certificate certs/fullchain.pem;
                ssl_certificate_key certs/privkey.pem;

                ssl_session_cache shared:le_nginx_SSL:10m;
                ssl_session_timeout 1440m;
                ssl_session_tickets off;

                ssl_protocols TLSv1.2 TLSv1.3;
                ssl_prefer_server_ciphers off;

                ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";

                if ($scheme != "https") {
                        return 301 https://$host$request_uri;
                }

                location = /stub_status {
                        stub_status;
                }

                location / {
                        proxy_pass http://n1.opti.lan:8084;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                        proxy_set_header Host $host;

                        proxy_http_version 1.1;
                        proxy_set_header Upgrade $http_upgrade;
                        proxy_set_header Connection "upgrade";

                        client_max_body_size 20G;
                }
        }
}

[stefan0xC]

Curious that it tries to load api.bitwarden.com. Can you inspect (or just clear) your local storage for that site? My current guess would be that you have somehow set the region to US but not sure how this would be possible. Do or did you maybe have a custom build web-vault at one point? Or maybe you have extensions installed that are interfering? Could you try with another browser?

[brandon1024]

Indeed. When I looked into this the other day, I cleared my local storage a few times without success. No references to bitwarden in my cookies or local storage either.

Do or did you maybe have a custom build web-vault at one point?

Nope :/

I tried this again today and I can't seem to reproduce the behaviour I was seeing. Odd. Let's park this discussion for now and I'll dive a bit deeper when the issue presents itself again.

[murtuzaakhtari]

I have been having this issue since a very long time as well. I am currently on vaultwarden 1.30.1 and vaultwarden-web 2023-12-0.

I have noticed that if I login, then refresh and put the password in one more time, I can see all the items. Now if you either lock your vault or let it lock on it's own, but don't close the browser tab and open another tab and login to the vault, all the items show up on the first go without having to refresh. However, if no other previous tabs for the vault are open, then the first login shows 0 items until you refresh. @brandon1024 , would you please test this on your end to confirm?

I do use HAProxy on my OpnSense firewall as a reverse proxy for vaultwarden. I have tried clearing the local cache for vaultwarden without luck. I have tried using Incognito/Private Windown & I have even tried downloading and using a completely different browser to no avail.

A fix for this would really be appreciated.

EDIT: My install is on an Archlinux LXC container on my Proxmox server using the packages from the Archlinux repos. I used to build my own until Archlinux started providing the vaultwarden packages in the official repos and then I switched to using the pre-built packages provided by my distro of choice.

[brandon1024]

@murtuzaakhtari great analysis, it looks like you went a bit deeper on this than I did. This behaviour still shows up every once in a while for me, it's a bit annoying. I still haven't found a root cause. It's good to see it's not just me running into this at least.

I'll make a note to dig into this a bit deeper next time it happens.

[murtuzaakhtari]

Thanks. Let me know if you find something else specific to this issue and a possible fix.
Here's another reddit thread describing the same issue, so I don't think we are the only ones facing this.

[Gerardv514]

Curious if you see this happening after doing an update? I too have noticed this in the past, although it is very very rare. When I do notice it, it is usually after doing an update. I have seen this on browser extension and also web vault. To resolve I would hit log out, close the browser and then log back in (same for the browser extension) and all would be fine. I have never tried hitting the refresh button though, will try that next time I see it to determine if both experiences are related.

[brandon1024]

It's quite rare for me too, maybe once or twice a month?

I just updated vaultwarden right now and cleared caches, couldn't reproduce. It's quite a tricky one to reproduce reliably.

[murtuzaakhtari]

I am able to reproduce this every time (provided no other tab is open for vaultwarden). Oh and I don't have any issues on the Chrome Extension or the Firefox extension. Those have never given me any problems. Just the web-vault.

[murtuzaakhtari]

I get the same errors in the Dev console as @brandon1024 .
I am not sure why it is trying to connect to api.bitwarden.com for a self-hosted install.

[brandon1024]

@murtuzaakhtari out of curiosity, do you have the bitwarden extension installed/enabled?

My hypothesis is that the Bitwarden extension injects content scripts into the page and those scripts are generating the errors we're seeing. I didn't try disabling the extension, but I should have tried that. If you're able to reproduce this relatively easy, could you give that a try?

[murtuzaakhtari]

Yet another test:

1. I closed all vaultwarden tabs
2. I opened a new tab for vaultwarden and logged in
3. I see 0 items, but instead of refreshing, I went to the "Accounts Settings" page
4. I see that it is trying to retrieve the account info from bitwarden.com and it fails to fetch because I don't have a Bitwarden account. After refresh, it then correctly tries to connect my personal domain name.
   

[murtuzaakhtari]

Sorry, saw your reply after posting mine. But yes, I do have the extension installed and enabled -- because it's convenient to simply Ctrl + Shift + L for any website you are trying to login to.

I'll try disabling it temporarily and see, but I don't think I would want that long term for obvious reasons.

[murtuzaakhtari]

I disabled the extension, restarted Chromium for good measure and tried it again, I still see 0 items on first login :(

[brandon1024]

@murtuzaakhtari awesome!

And yeah just disable it temporarily to see if it's any injected content scripts are causing the problem. If that's the case, we'll know roughly where to apply a fix.

Edit: Ahh bummer. Ok. Back to the drawing board 😅

[murtuzaakhtari]

One more test:

1. Opened a new tab for Vaultwarden
2. Logged in and I see 0 items.
3. Instead of refresh, I use the "Lock Now" option from within the user profile drop down.
4. Enter my password again and I still see 0 items. The Dev Console still seems to be connecting to bitwarden.com
   

I guess that only a Refresh(F5) is clearing out the bitwarden domain name and replacing it with the self-hosted domain name. I hope that helps the devs to at least get a start on finding the root cause.

[murtuzaakhtari]

Ok, I see what you mean now.

No, I have the admin panel disabled, so I don't have any config.json file. I only have the .env file which enables the web-vault and defines the location of the DATA_FOLDER.

[Gerardv514]

I guess others here would have way more of an idea than I do, but if domain is not set could this cause this kind of issue?

[BlackDex]

The domain has nothing to do with it, since the reddit post is talking about vault.bitwarden.com, it probably is not related to Vaultwarden specifically.

[Gerardv514]

I didn’t look into Reddit post since anything referring to that should be addressed there.

Was referring to the above unique scenario. I for sure cannot reproduce the above and is definitely different that my rare occasion where I have to fully log out and re log back in to get vault items back. In my experience it usually occurs after updating the container so probably has to do with the session.

In the above users scenario it’s able be reproduced on demand every single time. I believe there is something with the specific setup, I just wouldn’t know where to look since this is above my head; you guys are more the experts. But to go to your vault and it’s calling bitwarden’s domain rather than yours, and then after a refresh it then directs to your domain or api or w/e it’s called. There’s got to be something in the specific setup that’s causing this. This is a unique case and the majority does not have this problem. Maybe it’s something with the reverse proxy,
Or dns not sure.

Hope it can be found though, I’ve been running rock solid.

[murtuzaakhtari]

The domain has nothing to do with it, since the reddit post is talking about vault.bitwarden.com, it probably is not related to Vaultwarden specifically.

@BlackDex , yeah I have a self hosted install which is why I was asking why it was connecting to *.bitwarden.com instead of connecting to my local install. Again after a refresh, everything works as expected.

[stefan0xC]

Since it works after a reload my guess would be that is caused by race condition in the web vault somewhere. Maybe in the InitService where the base url is set because it looks like the web vault falls back to the US urls. I thought the bug was already fixed with bitwarden/clients#5801 but this might just be for selfhosted Bitwarden installations(?) and our changes reintroduce the issue somehow. Alternatively (especially if this issue still occurs also for selfhosted Bitwarden instances) this previous bug fix might not have addressed the root issue or it could also be caused by something else entirely. We might have to wait for this draft PR to be completed.

[zacknewman]

Since it works after a reload my guess would be that is caused by race condition in the web vault somewhere. Maybe in the InitService where the base url is set because it looks like the web vault falls back to the US urls. I thought the bug was already fixed with bitwarden/clients#5801 but this might just be for selfhosted Bitwarden installations(?) and our changes reintroduce the issue somehow. Alternatively (especially if this issue still occurs also for selfhosted Bitwarden instances) this previous bug fix might not have addressed the root issue or it could also be caused by something else entirely. We might have to wait for this draft PR to be completed.

I've been too lazy to debug the problem, but I have noticed this issue on the iOS client. I have a test server I deploy changes to; and when I change the server the app connects to, I have to close the app and reopen/unlock it to access the data as even explicitly syncing doesn't solve the problem.

In my specific case I use the same URL for both the test and production server and instead change the DNS AAAA record. I suspect that's not relevant though.

[Mik4sa]

One of my users also get this error. About once a day. We don't really have a clue what this is. Is there anything I can do to get more details about the error?

[murtuzaakhtari]

Hi @Mik4sa , can you have your user verify what I am seeing -- as in if they close all the Vaultwarden tabs, do they see 0 items in the subsequent login?

[DarkChemicalDC]

It is most likely to do with reverse proxy setup.
I was noticing simular behavior on my private instance (haproxy with custom acl's and TLS)
Debuging the problem from browser might not be possible. I would suggest enabling full logs on proxy. And inspecting what is shown there.
In my instance there where some 404 and 503 errors when responding to request.
I don't recall what precisely resolved the issue for me, but looking at the haproxy logs helped to troubleshoot.

Remember, browser might not show what it can't see.

[murtuzaakhtari]

Thanks, I will investigate and post back if I can figure it out.

[Gerardv514]

Anything further on this?

[N4SoftwareNinja]

I have the same problem nearly every day when I first login to our selfhosted vaultwarden web interface.

[brandon1024]

I haven't seen this issue in quite some time. Interestingly, this seems to coincide with a migration from slower hardware (raspberry pi 4) to faster hardware (dell optiplex). @stefan0xC you mentioned that this could be caused by a race condition; do you think this could be a race condition on server side or client side?

In any case, enough people have run into this that I think it's a good time to reopen the bug: #3781

[stefan0xC]

No idea because I have not run into this issue myself and have not found a way to replicate it. From the linked issues I would have assumed that it might be a client side issue which we cannot fix. (Could have been fixed already since they changed how/where the base URL is set.)

Are you still affected by this issue? Can you provide more information?

[murtuzaakhtari]

I am still getting this on every login. It's still trying to connect to bitwarden.com on first call. Refresh fixes it. No change in the past 4 months. Unfortunately, though, I haven't been able to triage it much further.

Here's my HAProxy settings for vaultwarden

#- ACL: vaultwarden
   acl acl_63276269c65d47.19509789 hdr_reg(host) -i ^[bit|vault]+warden\.MYFANCYDOMAIN\.com$

#- ACTION: vaultwarden
  use_backend vaultwarden if acl_63276269c65d47.19509789


#- Backend: vaultwarden ()
backend vaultwarden
   # health checking is DISABLED
  mode http
  balance source
  # stickiness
  stick-table type ip size 50k expire 30m  
  stick on src
  http-reuse safe
  server vaultwarden 192.168.5.25:8000 

[BlackDex]

Did you tested the latest testing tagged images already?
Those contain a newer web-vault.

[murtuzaakhtari]

I have not tested those yet.

[DarkChemicalDC]

What does Haproxy logs show?

[murtuzaakhtari]

I am using the HAProxy plugin on OpnSense and the log file doesn't show anything at all related to accessing vaultwarden except when a power outage had shutdown all my LXC containers on the Proxmox server:

[DarkChemicalDC]

apr 21 05:01:27 <server namer> haproxy[465909]: <ip addr clientr>:51630 [21/Apr/2024:05:01:27.905] http_in~ vaultwarden/vaultwarden 0/0/2/2/4 200 2677 - - ---- 6/6/1/1/0 0/0 "GET /vaultwarden/ HTTP/1.1"
apr 21 05:01:27 <server namer> haproxy[465909]: <ip addr clientr>:51630 [21/Apr/2024:05:01:27.985] http_in~ vaultwarden/vaultwarden 0/0/0/1/1 200 1916 - - ---- 6/6/1/1/0 0/0 "GET /vaultwarden/theme_head.1df11f603fda400762b7.js HTTP/1.1"
apr 21 05:01:28 <server namer> haproxy[465909]: <ip addr clientr>:51630 [21/Apr/2024:05:01:28.058] http_in~ vaultwarden/vaultwarden 0/0/0/2/124 200 267777 - - ---- 10/10/3/3/0 0/0 "GET /vaultwarden/app/main.bfda25c788b32075b928.css HTTP/1.1"
apr 21 05:01:28 <server namer> haproxy[465909]: <ip addr clientr>:51632 [21/Apr/2024:05:01:28.181] http_in~ vaultwarden/vaultwarden 0/0/1/4/159 200 220815 - - ---- 10/10/3/3/0 0/0 "GET /vaultwarden/app/polyfills.553f1c6e04a4256f1a4c.js HTTP/1.1"
apr 21 05:01:28 <server namer> haproxy[465909]: <ip addr clientr>:51635 [21/Apr/2024:05:01:28.485] http_in~ vaultwarden/vaultwarden 0/0/1/2/3 200 6913 - - ---- 10/10/3/3/0 0/0 "GET /vaultwarden/images/logo-dark@2x.png HTTP/1.1"
apr 21 05:01:28 <server namer> haproxy[465909]: <ip addr clientr>:51630 [21/
apr 21 05:01:28 <server namer> haproxy[465909]: <ip addr clientr>:51635 [21/Apr/2024:05:01:28.565] http_in~ vaultwarden/vaultwarden 0/0/0/1/2 200 77561 - - ---- 11/11/3/3/0 0/0 "GET /vaultwarden/fonts/bwi-font.f7a43f81d823e54532d4.ttf HTTP/1.1"
apr 21 05:01:28 <server namer> haproxy[465909]: <ip addr clientr>:51638 [21/Apr/2024:05:01:28.595] http_in~ vaultwarden/vaultwarden 0/0/1/1/2 200 56777 - - ---- 11/11/3/3/0 0/0 "GET /vaultwarden/fonts/Open_Sans-normal-400.8465030b62273715c99d.woff HTTP/1.1"
apr 21 05:01:28 <server namer> haproxy[465909]: <ip addr clientr>:51633 [21/Apr/2024:05:01:28.182] http_in~ vaultwarden/vaultwarden 0/0/0/4/746 200 2358488 - - ---- 11/11/2/2/0 0/0 "GET /vaultwarden/app/main.a8f027a034b348c3f19d.js HTTP/1.1"
apr 21 05:01:29 <server namer> haproxy[465909]: <ip addr clientr>:51634 [21/Apr/2024:05:01:28.182] http_in~ vaultwarden/vaultwarden 0/0/1/3/1083 200 2519645 - - ---- 11/11/1/1/0 0/0 "GET /vaultwarden/app/vendor.58a71e94f36c45876e9f.js HTTP/1.1"
apr 21 05:01:29 <server namer> haproxy[465909]: <ip addr clientr>:51634 [21/Apr/2024:05:01:29.724] http_in~ vaultwarden/vaultwarden 0/0/0/2/7 200 226871 - - ---- 11/11/1/1/0 0/0 "GET /vaultwarden/locales/en_GB/messages.json?cache=mw0sn HTTP/1.1"
apr 21 05:01:29 <server namer> haproxy[465909]: <ip addr clientr>:51634 [21/Apr/2024:05:01:29.790] http_in~ vaultwarden/vaultwarden 0/0/0/2/2 200 4690 - - ---- 11/11/1/1/0 0/0 "GET /vaultwarden/images/apple-touch-icon.png HTTP/1.1"
apr 21 05:01:29 <server namer> haproxy[465909]: <ip addr clientr>:51633 [21/Apr/2024:05:01:29.795] http_in~ vaultwarden/vaultwarden 0/0/0/2/2 200 2325 - - ---- 11/11/2/2/0 0/0 "GET /vaultwarden/images/favicon-32x32.png HTTP/1.1"
apr 21 05:01:29 <server namer> haproxy[465909]: <ip addr clientr>:51630 [21/Apr/2024:05:01:29.794] http_in~ vaultwarden/vaultwarden 0/0/0/2/9 200 226765 - - ---- 11/11/1/1/0 0/0 "GET /vaultwarden/locales/en/messages.json?cache=mw0sn HTTP/1.1"
apr 21 05:01:29 <server namer> haproxy[465909]: <ip addr clientr>:51634 [21/Apr/2024:05:01:29.874] http_in~ vaultwarden/vaultwarden 0/0/0/1/1 200 1839 - - ---- 11/11/1/1/0 0/0 "GET /vaultwarden/api/config HTTP/1.1"
apr 21 05:01:29 <server namer> haproxy[465909]: <ip addr clientr>:51634 [21/Apr/2024:05:01:29.898] http_in~ vaultwarden/vaultwarden 0/0/0/2/4 200 59197 - - ---- 11/11/1/1/0 0/0 "GET /vaultwarden/fonts/Open_Sans-normal-600.7f2861051b34a8ecd9f2.woff HTTP/1.1"

This is how initial connection to vaultwarden looks like in haproxy logs.
I am not aware how to enable full logs in OpenSense implemitation, because I use generic x86_64 for linux.
If you see something else than 200's respones from haproxy you are configured wrong.

My setup also uses that the vaultwarden instance is hosted on /vaultwarden to make things even more hard. Everything works.
Either OpenSense implemention is wrong, or your setup is wrong.

[Raz0rmob]

We apparently have the same problem. Some users report that when they log in, the safe is empty. Only after refreshing or logging in again do they see the content.

[BlackDex]

I'm still puzzled to why your browsers want to connect to bitwarden.com instead of your own instance.
That shouldn't happen at all, and that is what is causing these weird issues i think. I'm just not sure where that is coming from.

[darrengoldner]

Adding to this, I can reproduce fairly reliably as well and it does seem like some sort of race condition where the web vault's variables are not instantiated so there's a fallback
Console log on login:

Lock screen manifestation doesn't have web vault domain variable __$2__

[Raz0rmob]

When will the bug fix be implemented in the stable version?

[BlackDex]

Tested the testing image already to verify it solves the issue?

[Raz0rmob]

We do not want to use the test image for our production system, and a second instance is not planned. However, I assumed that Anarute and Pabelenda had already tested it because of the thumbs-up emote.

[jmallach]

We have just updated to 1.31 and so far we haven't seen the bug manifest again. We'll keep an eye on the following days and will report back.

[murtuzaakhtari]

Currently using vaultwarden 1.31.0 and web-vault 2024.5.1 and have not encountered the issue yet. Will keep monitoring it for the next few weeks to be absolutely sure the issue is fixed.

[brandon1024]

I totally forgot about this issue -- I haven't seen it in some time either, it started working again around ~1.30.0-1.31.0. Closing this discussion 👍