Performance Issues #1453
Comments
|
Please supply more information preferably by using the |
Updated with more info thanks, unfortuantly the button didn't generate anything. |
|
Im still missing some logs, amount of ciphers etc.. we can't do anything with this. There is nothing we can use as a point of reference. As a side note, what was the error when trying to generate the support string? |
The button didn't function for some reason, with no error. I'm not sure what the issue was, but I've now managed to generate the support string. Please have a look at the OP, thanks. |
|
Well it all looks ok. Also don't think the amount of vault items it's very much. Could you provide some logs when performing the actions which takes a lot of time?
|
|
I've monitored the logs, and there's nothing unsual showing. It's weird, because the task occurs, and the page loads after a very long period of time, however the interface just becomes unresponsive/shows blank screen when you bulk delete/add entries. I'm not sure if this is intended behaviour, but it takes an awfully long time. The only thing the log shows is: |
|
I do not see log lines regarding deletion here. |
|
Here's a screenshot, however I didn't manage to capture the freeze as it happens on random occassions. https://i.imgur.com/HM2H0ai.png I will try to monitor when it does hang, and will update you with the results. Thanks |
|
That image is not readable so i can't determine where to start looking. |
You should be able to view the screenshot clearly by zooming in, or do you mean it doesn't provide much info? I'm using bitwarden_rs as an Home Assistant addon on Ubuntu, and the logs in portainer/docker/addon do not record any data to show this info. Also, I'm not able to find the |
|
This seems unlikely to be a bitwarden_rs issue. More likely it's something about your reverse proxy or networking configuration, or the way your addon has configured that. I'd suggest you take this issue to https://github.com/hassio-addons/addon-bitwarden. |
|
Hello, I am having some performance issues as well. Here is a small chunk of logs showing the issue # If you look at the request that started at Support String: Your environment (Generated via diagnostics page)
Config (Generated via diagnostics page){
"_duo_akey": null,
"_enable_duo": false,
"_enable_email_2fa": true,
"_enable_smtp": true,
"_enable_yubico": true,
"_ip_header_enabled": true,
"admin_token": "***",
"allowed_iframe_ancestors": "",
"attachments_folder": "data/attachments",
"authenticator_disable_time_drift": false,
"data_folder": "data",
"database_max_conns": 10,
"database_url": "****/**.*******",
"db_connection_retries": 15,
"disable_2fa_remember": false,
"disable_admin_token": false,
"disable_icon_download": false,
"domain": "*****://***.************.***",
"domain_origin": "*****://***.************.***",
"domain_path": "",
"domain_set": true,
"duo_host": null,
"duo_ikey": null,
"duo_skey": null,
"email_attempts_limit": 3,
"email_expiration_time": 600,
"email_token_size": 10,
"enable_db_wal": true,
"extended_logging": true,
"helo_name": "*********",
"hibp_api_key": null,
"icon_blacklist_non_global_ips": true,
"icon_blacklist_regex": null,
"icon_cache_folder": "data/icon_cache",
"icon_cache_negttl": 259200,
"icon_cache_ttl": 2592000,
"icon_download_timeout": 10,
"invitation_org_name": "Bitwarden @ **********",
"invitations_allowed": false,
"ip_header": "X-Real-IP",
"log_file": null,
"log_level": "Info",
"log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
"org_attachment_limit": null,
"org_creation_users": "",
"password_iterations": 100000,
"reload_templates": false,
"require_device_email": false,
"rsa_key_filename": "data/rsa_key",
"show_password_hint": true,
"signups_allowed": false,
"signups_domains_whitelist": "",
"signups_verify": false,
"signups_verify_resend_limit": 6,
"signups_verify_resend_time": 3600,
"smtp_accept_invalid_certs": false,
"smtp_accept_invalid_hostnames": false,
"smtp_auth_mechanism": "Plain",
"smtp_debug": false,
"smtp_explicit_tls": false,
"smtp_from": "**-*****@************.***",
"smtp_from_name": "Bitwarden @ *****",
"smtp_host": "**.************.***",
"smtp_password": "***",
"smtp_port": 587,
"smtp_ssl": true,
"smtp_timeout": 2,
"smtp_username": "*****@****.************.***",
"templates_folder": "data/templates",
"use_syslog": false,
"user_attachment_limit": null,
"web_vault_enabled": true,
"web_vault_folder": "web-vault/",
"websocket_address": "0.0.0.0",
"websocket_enabled": false,
"websocket_port": 3012,
"yubico_client_id": null,
"yubico_secret_key": null,
"yubico_server": null
} |
BlackDex
added
the
troubleshooting
|
SQLite DB is not the best on performance side, do you reproduce on mysql/postgresql? |
|
I will have to figure out how to switch to postgres. I have a postgres instance running already with other schemas, would that be a security risk, given items in Bitwarden are end to end encrypted? |
|
As password managers aren't write-intensive workloads, SQLite will likely perform better than MySQL/PostgreSQL for most users. BTW, this is not clearly a bitwarden_rs issue, so you should take this to the forums. |
BlackDex
added
enhancement
|
Another performance problem here, I met now. Cloudflare continously give me 524 error after importing hundreds of password from 1p. After check the discussion here: #1470 I think things are not that clear. I changed the worker number via ENV when running with docker, the workers are limited to 4 concurrently. However, the log reveals that The web page were trying to downloading all the favicon of the items recorded in my account, since today my server network is under a heavy load, it becomes slow, Then finally I personally recommend two solution for my situation, either should be good:
I personally like the second one. BTW, I'll increase the worker number and have a try. |
|
@kmahyyg, well the first option is something you should request at upstream bitwarden since that is a client item And the second is already built into all the clients. You can disable icon downloading per client in your user settings. You can also disable icon downloading on the server side, but that does not prevent the client from downloading. |
|
I've met a similar issue: after editing an item, there is a chance to trigger batch update of icons, which will freeze the server.
To resolve the issue I configured nginx to use CDN for icon downloading: |
|
For those running Vaultwarden on Kubernetes this ingress example solved the crashes for me (using ingress-nginx). Thanks @lanceliao This rewrites all requests to |
I have also experienced this after switching from the old bitwarden_rs container to the vaultwarden container. |
|
Hi, I almost forget this issue here. First, I switched to MySQL when deploying to the production environment. Second, I disabled favicon auto-fetching on both client and server. Since I hate nginx, disabling favicon is the best option I have. (And I really do not need any favicon since all my login items are well organized) |
|
People having issues with icon requests preventing Rocket from servicing other API requests can consider setting an external icon service (#2158). This is similar to redirecting icon requests via reverse proxy, but implemented within Vaultwarden. This feature is currently only available in the |
|
Since #1545 (which I wrote to solve a very similar issue), the browser should be caching icons like this, for a very long time, meaning no requests should be hitting the server at all. It appears it's actually the browser causing the issue. When loading the vault after logging in, the browser makes all the icon requests for all entries. If you wait for those to complete, switch to the "Send" tab and then come back, the images are correctly cached and no extra requests are made. Which is odd! Unfortunately, throwing a bunch of requests at a thread-based server will cause it to lock up. Once the port to Rocket's async implementation comes in, that will absolutely help with all this. In the mean time using an external service should work, although personally I'd really rather not do that. Alternatively, cranking the number of rocket worker should help, or putting a cache in nginx instead, as that's more suited to handling this many concurrent requests. It's possible using some different caching methods, along with actually lazy-loading the images will make a huge difference. I'll have a play around this afternoon, see what comes out... |
|
I've opened a change upstream which should help with that influx of icon requests. On the one hand it's great to improve the application performance, and cache the responses, but not making said requests in the first place will likely help out even more! bitwarden/jslib#591 |
|
If you wonder how to do this with Caddy reverse proxy, adding this to my server config seems to work just fine for me: The |
That isn't needed at all anymore since this is already a build-in option. |
|
Thanks, didn't know this already landed in a release. |
|
Vault Items: 921 I have done some testing with different kind of data. Tested with url fields and without. I do not think that the performance issues are related to the icon loading. Tested the web vault loading with: I think you can see where this is going. To more ciphers there will be the slower the web vault will become. |
|
now we have |
|
What is load time exactly btw? |
|
So to add some more details. The PostgreSQL database host is hosted in the same data center (latency <1ms). We did enable slow query logging. We saw that all the DB queries happen very fast. With the load time I mean when you login to Vaultwarden, refresh the page or go to settings and then back to the organization. I have the developer tools open, network tab and there you can see "/api sync?excludeDomains=true" takes ~32s. It is the same when we use the python module. It is the same for the official Bitwarden client, once you logout and login again, it will take the 32s to load. If you close the client or lock, it will load instantly. For the official client it seems that it clears the cache once you logout. |
|
@BlackDex, hi, I'm colleague of @perkons so will answer questions on his behalf and provide more details. Logging in into the Vaultwarden web-ui happens quickly, almost instantly, but then it stays like this for about 30 seconds before the list of our passwords appear: By looking at at the browsers developer tools, I can see that it takes this long to open Other than loading initial list of passwords vaultwarden interface responses are quick and there are no issues. RTT between my laptop with browser to vaultwarden is about 7ms. We have also some scripts that are using python bitwardentools module to add and retrieve passwords and they are also experiencing same performance issues while doing the sync, to show you an example: Also, we enabled SQL query performance logging on our PostgreSQL server and can see that all vaultwarden SQL queries are executed very quickly, maximum time for the SQL query execution that we've seen was around 10ms, so we can exclude an issue with SQL backend performance. |
|
I'm not sure which version of Vaultwarden you are running. But it would like to know if running the latest I do suggest to create a backup of the database (Or maybe even clone it and run a different container). |
|
Vaultwarden version: 1.24.0 |
|
@BlackDex, we're running 1.24.0-alpine image, I will update our dev environment to testing to see if that helps And I also forgot to previously mention, that we've tried enabling debug logging for vaultwarden, but that didn't produce any useful logs about this issue. |
|
@karlism Nice. A few notes on the testing image and why it could be helpful with speed.
|
|
@BlackDex, thanks for your help! I've updated it to latest testing image: Unfortunately I do not see any difference as This is output of the |
|
Well, the main thing is loading those entries and converting them to json. The strange thing is, on my test environment i have a lot more items, but it loads a lot faster. Also running that test script i get the following result: time python3 vaultwarden-sync.py
before sync: 14/04/2022 15:48:25
after sync: 14/04/2022 15:48:25
real 0m2,018s
user 0m0,227s
sys 0m0,030sI do use sqlite here, i would need to test a postgresql an other time |
|
@BlackDex, can you please share items from your test enviroment so we can try it out here? Or we can just try generating CSV list of 2,000 passwords like these ourselves: Just to exclude the possibility that it's something related to our passwords. |
|
bitwarden_org_export_test-data.csv.zip Attached is a list i have loaded into my test vault (With a bunch of other items not listed there). It does take a long time for them to be imported. It may even happen that it reports a gateway timeout, but those items will still be loaded into the database, but again that could take a few minutes maybe. I still need to create a good json file for this to really test all the separate features and items. But until now not really needed. Might do that this weekend maybe |
|
MySQL/PostgreSQL performance issues are more likely related to network latency, as in #1402. You mention you have less than 1 ms latency, which may not sound like much, but since Vaultwarden uses a lot of small queries, this can really add up. This model is something an in-process database like SQLite excels at (unless its database file is placed on network storage). If you try running a PostgreSQL server on the same machine as Vaultwarden, I expect you would see much better performance. N.B. Upstream Bitwarden uses a lot of stored procedures to address performance issues like this, but it has its own drawbacks, like a lot more maintenance burden. |
|
Regarding this issue and #1402 i just did some logging of all the queries done during a sync. I definitely think there is room for improvement. During the query logging i saw that some queries which are exactly the same are run as much times as there are items within an organization. We might be able to cache those for like 30s or i would even like to prevent them from running multiple times at all. Ill see if i have some time to take a better look at it. But it could take some time, since this is not very easy i think with all the dependencies of all functions and conversion from/to json etc.. |
|
Sorry, but I've previously misled you, after further testing I noticed that database and pod, where Vaultwarden is running, were in different datacenters (with 6-7ms latency in between them). After moving pod to the same datacenter, where database is running (and latency is below 1ms), the problem resolved and 921 password load time dropped from 22 seconds to below 1 second. @jjlin was spot on regarding database latency impact on Vaultwarden performance. |
|
Nice to see that it solved that issue. |
Improved sync speed by resolving the N+1 query issues. Solves dani-garcia#1402 and Solves dani-garcia#1453 With this change there is just one query done to retreive all the important data, and matching is done in-code/memory. With a very large database the sync time went down about 3 times. Also updated misc crates and Github Actions versions.
Improved sync speed by resolving the N+1 query issues. Solves dani-garcia#1402 and Solves dani-garcia#1453 With this change there is just one query done to retreive all the important data, and matching is done in-code/memory. With a very large database the sync time went down about 3 times. Also updated misc crates and Github Actions versions.
|
Solved via #2429 |
Describe the Bug
When removing/adding entry on the web Vault, everything because super laggy, and unresponsive.
I have had the page completely freeze with a white screen, the only way to access the page again was to refresh it, which cancels out the current actions.
Steps To Reproduce
Delete/edit any item on the web Vault.
Expected Result
Be responsive, and not freezes the page.
Actual Result
Becomes unresponsive, and freezes.
Environment
Config (Generated via diagnostics page)
{ "_duo_akey": null, "_enable_duo": false, "_enable_email_2fa": false, "_enable_smtp": true, "_enable_yubico": true, "_ip_header_enabled": true, "admin_token": "***", "allowed_iframe_ancestors": "", "attachments_folder": "/data/attachments", "authenticator_disable_time_drift": false, "data_folder": "/data", "database_max_conns": 10, "database_url": "/****/**.*******", "db_connection_retries": 15, "disable_2fa_remember": false, "disable_admin_token": false, "disable_icon_download": false, "domain": "*****://*****.*******.***", "domain_origin": "*****://*****.*******.***", "domain_path": "", "domain_set": true, "duo_host": null, "duo_ikey": null, "duo_skey": null, "email_attempts_limit": 3, "email_expiration_time": 600, "email_token_size": 6, "enable_db_wal": true, "extended_logging": true, "helo_name": null, "hibp_api_key": null, "icon_blacklist_non_global_ips": true, "icon_blacklist_regex": null, "icon_cache_folder": "/data/icon_cache", "icon_cache_negttl": 259200, "icon_cache_ttl": 2592000, "icon_download_timeout": 10, "invitation_org_name": "Bitwarden_RS", "invitations_allowed": true, "ip_header": "X-Real-IP", "log_file": null, "log_level": "Info", "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f", "org_attachment_limit": null, "org_creation_users": "", "password_iterations": 100000, "reload_templates": false, "require_device_email": false, "rsa_key_filename": "/data/rsa_key", "show_password_hint": true, "signups_allowed": true, "signups_domains_whitelist": "", "signups_verify": false, "signups_verify_resend_limit": 6, "signups_verify_resend_time": 3600, "smtp_accept_invalid_certs": false, "smtp_accept_invalid_hostnames": false, "smtp_auth_mechanism": null, "smtp_debug": false, "smtp_explicit_tls": false, "smtp_from": "", "smtp_from_name": "Bitwarden_RS", "smtp_host": null, "smtp_password": null, "smtp_port": 587, "smtp_ssl": true, "smtp_timeout": 15, "smtp_username": null, "templates_folder": "/data/templates", "use_syslog": false, "user_attachment_limit": null, "web_vault_enabled": true, "web_vault_folder": "web-vault/", "websocket_address": "0.0.0.0", "websocket_enabled": true, "websocket_port": 8080, "yubico_client_id": null, "yubico_secret_key": null, "yubico_server": null }The text was updated successfully, but these errors were encountered: