MSSQL Backup Issues #856
Comments
|
I suggest that you write a support ticket here: https://bitwarden.com/contact/ so you can share with us your logs and all the modifications that you have done to the docker-compose and docker-compose-override yaml. |
|
The only logs that I can see regarding the error are pasted above, but I have emailed what you've requested through and explained why those settings have been set. |
|
This likely has something to do with the filesystem environment. Mysql on docker is a bit flakey with filesystems and only works well under ext4 as far as I know. Otherwise, you need to use a docker volume as you have discovered. |
|
Seems strange for it to have failed all of a sudden after 3 months of zero changes. In addition, while the docker volume works for gaining access to my database, the auto backups are failing... which leaves me somewhat uneasy in the event I lose my databases again for corruption or some such. I'm hoping that's also not filesystem related? |
|
I don't think the backup volume is mapped to the docker volume, so you would likely need to change that also using a docker-compose modification or override. |
|
When I swapped the docker database volume from false to true, the yaml changed to include volumes and added this to the bottom of the docker-compose yaml: I didn't fill in anything after mssql_data: at the bottom, because the same parameter seems to be prefilled already under bitwarden/mssql: To me, that suggests it should be mapped? Or are you saying the second field needs to be manually specified? |
|
You also need to pay the backups to a docker volume: This: And probably logs too. |
|
So, you're saying replacing the existing volumes of: To something like: Or am I interpreting that incorrectly? |
|
Change it to something like this: |
|
Ok, I can give that a go. Can you elaborate why mssql_data needs to be defined three times though? I thought the existing logs/mssql and mssql/backups were the only things changing? Am curious is all. EDIT: I've made the suggested changes to the bitwarden-mssql section of the docker-compose.yaml file, like so: If that is how you're asking to set it (sorry for the thousand questions), auto backup is due to trigger in ~40 minutes so we should know by then if it has worked or not. |
|
Hmm, it seems the changes made have caused the backup to not run at all. This is all that is available in the bitwarden-mssql log files (the backup folder is also no different). |
|
Ok, update to this... turns out the container was in UTC this whole time :\ which is why nothing happened when I said it would have. The logs suggest the backup WAS successful, but the file has not appeared in the backups folder on the host, only within the container. From Unraid host: From within the bitwarden-mssql container: @kspearrin would this be because the backups (and probably logs) folder is now mapped within the container only? Not pointing towards the host/docker volume? |
|
If you are using a docker volume, mssql_data, then the backup file is in there, not mapped back to your host. |
|
Well, as far as I can tell the mssql_data docker volume doesn't exist anywhere (not within the Bitwarden host folder, and not within the mssql container). I had a thought of working around it with an Unraid user script to copy the new files nightly, but docker doesn't support wildcard docker cp (been an outstanding request for 6 years +) as the name of the file changes per day. While the backups are now working, I can no longer access them without doing a docker cp from container to host, which isn't ideal. |
|
You may want to read more about how to interact with docker volumes on their website: |
|
Seems like I'm getting nowhere with this :( From what I've tried interpreting with that link, I have modified the docker-compose.yaml with the following: docker-compose-override.yaml: Failed this morning: The error sounds like write permission issues to the host, so I checked the backup folder and it has 777 permissions, but the owner is 65534:65534 - not exactly sure who or what that is, but it's a recurring theme across the bitwarden folders. From what I've read, Unraid dockers usually have issues if the owner is not set to nobody:users, so I've changed just the backup folder to that:
If it doesn't work, I'll change it back to the previous owner. Will have to wait for the nightly backup now, but if that doesn't work I guess I'll stick to mapping everything to mssql_data and docker cp from container to host every day, for lack of other ideas. |
|
Nope, still failed. I've reverted back to using mssql_data for everything. If that works, I'll leave it as is, even though it's not working as it should be. |
|
No longer working :( docker-compose.yaml (only the mssql part as the rest of the config file has been left untouched): docker-compose-override.yaml: This is how it was set when the backup worked once into the container (and only once). @kspearrin, any other ideas? |
|
What are the unlimits for? |
|
That was mentioned in the original email I sent via the Bitwarden contact form, but it's an Unraid requirement for running MSSQL on Unraid. Without ulimits defined, the bitwarden-mssql containers throws the following errors: See: https://forums.unraid.net/topic/82110-mssql-docker-on-unraid/ |
|
Maybe you are hitting these limits when backups run? |
|
The backups had run once during the course of time this issue was opened, but I don't believe that to be the case. I had increased them to double what they are currently: That didn't make a difference with the error. Is there any way to determine if that is actually the case though? (hitting the limits). The bitwarden-mssql container identifies 25716MB RAM during boot-up and the server itself has ~30TB of usable storage, so I don't believe the issue is physical RAM or storage. |
|
Just an update on this. Backups have been running successfully in the container for the last 6 days, but in container only. Can't get them to route to the host without causing backups to fail. Final setup:- docker-compose.yaml: docker-compose-override.yaml: The change in ulimit on its own didn't fix this unfortunately. It also required the mssql variables changed as well (each on their own caused backups to fail, for whatever reason both together allow the backups to work): As seen below: While I'd prefer the backups to point back to the host, this works and I'm no longer willing to break it. Closing ticket. Thanks @kspearrin for your help with this. |
Hi,
I'm having issues with MSSQL backups of my database no longer automatically backing up. Most recent error:
The last successful backup was last Friday (07/08), but wasn't aware of this until yesterday. Prior to this issue, my bitwarden-mssql container was constantly restarting due to this error:
Operating system error 317(The system cannot find message text for message number 0x%1 in the message file for %2.) encountered.After a few hours of troubleshooting, it was only fixed by setting the database_docker_volume parameter to true (was false) in the config.yml file. Changing that flag however made me lose access to the existing databases, so I ended up restoring the most recent backup following this guide: #651 and that allowed access to my password database again.
Come today and the backup error log above is now my new problem, which I am having issues fixing as I don't really know where to start or which logs to look into.
The bitwarden install sits on an Unraid environment (which as best as I can tell is probably not fully supported for how I've implemented it). I wasn't able to use bitwarden_rs from the CA App store as the installation kept throwing C# errors, which the dev wasn't sure why. I managed to get the official self-hosted instance of bitwarden working by making a few changes to the docker-compose and docker-compose-override yamls and this has been working perfectly fine for over 3 months.
TL;DR: Any idea why MSSQL is no longer backing up my database based on the error(s) above?
The text was updated successfully, but these errors were encountered: