Cannot delete LXD zfs backed containers: dataset is busy #4656
Labels
Comments
|
It's most likely some other process that's forked the mount table and is now preventing LXD from unmounting the container... You can run |
|
You mean lxd delete still reports dataset is busy :( Edit: grepping for the other, running, container results in lots of hits, so I think I have formatted that grep correctly. It seems there is nothing referencing test1 in proc/*/mountinfo. Any further ideas? :) |
|
Hmm, then it's not mounted anywhere visible which would likely make it a kernel bug... Sorry I don't have a better answer for this. |
|
@stgraber Oh dear Cthulhu, that's bad. That also explains why I'm seeing it across several systems, as I keep my servers in synch w/regards to kernel/os/package versions. I just checked on one of my other systems, and there I have a dataset which I still cannot destroy even after 48+ hours. So it does not seem this will go away on its own. There it is also "invisible". If you want access to the server Stéphane and poke around a bit, let me know. Otherwise I guess I'll just have to mitigate this manually (sigh) and update my kernels when I get the chance, and hope that resolves the issue. PS: I am not used to seeing grep issued that way, your command was of course correctly formatted, I just assumed it didn't since I didn't get any hits #n00b |
|
Should I report this as a bug somewhere, you think? |
|
If others stumble on this issue: There is a workaround in that it is possible to rename the dataset. So if your container is stopped, you can do:
After which you can issue
However you then still have this dataset hanging around, which you will need to clean up at a later date, i.e. after a reboot I suppose. This pretty much sucks! :D |
|
Yeah, that's pretty odd, I wonder what's keeping that active... You don't have any running Just run If not, then I'd be happy to take a look, see if anything stands out. There's a pretty good chance that it's a kernel bug, but we haven't seen reports of this before so it's intriguing. |
|
(Note that I'm on vacation in Europe this week so not quite as around as usual :)) |
|
Nope no zfs commands running. I have sent you access by mail :) - enjoy your vacation..!! |
|
Very weird. I poked around for a bit, eventually restarting the Now that we know that kicking lxd apparently unsticks zfs, can you let me know if you have another machine with the same issue (or can cause the one I already have access to to run into it again)? I'd like to see what LXD shows as open prior to being killed, then if just killing it is enough to make zfs happy and if not, then why would lxd starting again somehow unstick zfs. FWIW, what I tried before restarting lxd was:
None of which showed anything relevant... |
|
It could be an fd leak from a file that was read or written from the container by LXD and wasn't closed, but what's odd is that if it was the case, we should have seen an fd with the container path and there were none of them... Hopefully I can look at another instance of this problem and figure that part out. |
|
Marking incomplete for now, @Kramerican let me know when you have another affected system. |
|
I sure will @stgraber I'm also on vacation and haven't had a chance to check if I can provoke this behaviour. I'll let you know. |
|
@Kramerican still on vacation? :) |
|
@stgraber Yes until next week - but I will set some time aside then to try and force this behavior. |
|
@stgraber I have had little luck in forcing this behavior, but this has been cropping up all over the shop these last few days. I had written some fallback code in our tools which simply renames the dataset, so that lxc delete could be run. These datasets are still "stuck" and zfs refuses to delete them. I have not restarted lxd in order to delete them - is it enough for you to get access to one of these systems to diagnose further? In which case let me know and I'll give you access. Thanks..! |
|
@Kramerican yep, having access to one of the systems with such a stuck dataset should be enough to try to track down what LXD's got open that would explain the busy error. |
|
@stgraber Excellent. Mail with details sent. |
|
@Kramerican so the only potential issue I'm seeing is a very large number of mapped |
|
If it's an option at all, at least on your Ubuntu 18.04 systems, I'd recommend considering moving to the lxd snap (--channel=3.0/stable in your case) as that would get you a much faster turnaround for fixes than we can do with the deb package. |
|
@stgraber Excellent. However on the system where you have access, I have apt upgrade hanging at 95% at /var/log/lxd/lxd.log shows and entry which I think is responsible: Is raft a process I can kill? Suggestions on how to unstick the upgrade? |
|
@stgraber Nevermind - it got unstuck after a while. Everything seems fine. I will upgrade all systems and report back if the issue persists. Thanks..! |
|
@Kramerican pretty sure it got unstuck because I logged in and ran It's actually a bug that 3.0.1 fixes but if your database has too many transactions prior to the upgrade, it still fails to start. The trick to unstick it is to temporarily move it to a very fast tmpfs which I'm doing on that system now. |
|
@stgraber ah yes I saw that lxc ls and other commands were not working. I won't mess around on that system anymore until you report back. Series of commands which will help unstick lxd would be nice to have here, in case I see this happen on one of the other ~10 hosts I need to upgrade |
|
@Kramerican all done, that system is good to go. If you hit the same problem, you'll need to:
This will unstick the update. Once the update is all done, run again (for good measure):
That should ensure that LXD is fully stopped (containers are still running fine though).
You'll see the daemon start, let it run until it hits "Done updating instance types" which is when it'll be ready for normal operation, then hit ctrl+c to stop it. Once done, do:
And you'll be back online with the newly compacted and much faster database. |
|
This is only needed on systems where LXD isn't able to load its database within the 10s timeout so hopefully a majority of your systems will not need this trick. Once LXD successfully starts once on 3.0.1, the database gets compacted automatically in the background as well as on exit to prevent this problem from ever occurring again. |
|
@stgraber This is pure epic. Thanks so much, I'll get started cracks knuckles :) |
|
@Kramerican I've also deleted the two failed datasets on sisko, so restarting LXD did the trick to unstick zfs, now the question is whether we'll be seeing the issue re-appear with 3.0.1. |
|
@stgraber ok so I completed the upgrade on all my hosts, only had to follow your steps here on one other system 👍 In the process however it turns out that one of my systems was already all up to date with 3.0.1 and here the failure with a stuck dataset happened today I have just sent a mail with details and access to the system |
|
I'm going to see where the work tasks are being added that kick off the umount, that will give some insight. |
|
The exit of the lxc process via do_exit calls exit_task_namespaces -> switch_task_namespaces -> free_nsproxy -> put_mnt_ns -> drop_collected_mounts -> namespace_unlock -> mntput_no_expire adds __cleanup_mnt() to process' task work list, which in turn gets called asynchronously at the death of the lxc process. And this unmounts the zfs file system. |
|
We've now rolled out a fix for some, maybe even all of those issues. The issue that was clearly identified and now resolved in the current stable snap (latest or 3.18 track) was related to In this case, the way snapd manages mount namespaces was causing the zfs mount to be held by a forked of the mount namespace, preventing it from fully going away until all The fix is effectively a whole bunch more mount table magic now happening in the LXD snap. As part of this, I've also significantly improved logging for the mount logic in the snap package, so failures to reshuffle the mount table on snap startup will now result in errors being logged in |
|
Thanks @stgraber. I updated the LXD snap to version 3.18 (12181) and rebooted a couple of machines. In the first couple of days the problem seemed fixed, however here it is again: This is on my up-to-date Eoan laptop, but happens also on Bionic (GA kernel): Both the systems were rebooted after refreshing the LXD snap: This is now happening every time, as before. As I wrote in the Launchpad bug, the behavior I observe is the system switching from a "good state" (containers always get deleted successfully) to a "bad state" (the "dateset is busy" error is always hit). I can't tell what triggers the switch. [1] https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779156/comments/9 |
|
I collected some logs, as you mentioned that logging should now be more informative. So after doing Here is the output of And this after running |
|
@paride Can you get me:
I want to make sure that the mount table entries all match what we'd expect. |
|
Sure. Pastebin: https://paste.ubuntu.com/p/kRcbMR6XP6/ (but note that when I bootedI still had to set In what follows the container named
|
|
Now that's weird, there are no errors around failure to setup the mount propagation bits and the intermediate mounts are present, but the propagation settings aren't there... Can you show:
So hopefully I can get a test VM which mostly matches this to see why we would have the kernel tell us propagation was applied but then not have it actually be set. |
|
On the Eoan machine: On the Bionic machine: |
|
@paride ok, the current stable snap should have a fix for that one. |
|
I'm testing it, so far so good, but let's give it a few days before declaring the issue fully solved. |
|
@paride still good on your side? |
|
@stgraber, let me know if you need anything else. |
|
@smoser the mount propagation intermediate mount appears missing in your case which would explain the behavior. Did you reboot your system at some point after Oct 16th? If not, could you reboot, start a few containers and get me the output of In theory, stopping all containers and restarting LXD may also achieve most of the same effect, though having a clear starting point with the new logic in place would certainly be better. |
|
I have not rebooted since the 16th. So I'll try rebooting tomorrow and try to remember to report back here. thanks for taking a look. |
|
@stgraber after a week, still good. From my point of view the issue looks fixed. Thanks a lot! |
|
Well... i rebooted, and survived a which was failing yesterday. It wasn't 100% recreate previously, but when it got into a bad state it would stay there. I'll keep fingers crossed. |
|
@smoser can you show that mountinfo data? ( |
|
|
@smoser good, so you do have the intermediate mount in there now and your containers are all properly underneath it, so that should work as expected. Tentatively closing this issue. |
|
This is excellent news. Thanks for solving this @stgraber. |
paride
referenced
this issue
in paride/pycloudlib
Sep 2, 2020
The two-step (stop, delete --force) LXD container deletion was introduced in 3ec4cc2 to workaround the "ZFS dataset is busy" LXD bug [1] which was triggered by force-deleting running containers. We now drop the workaround because: 1. It's not needed anymore (the bug got fixed). 2. It was itself buggy: ephemeral containers are deleted when stopped, thus the "lxc delete" step failed because the container is gone already. [1] https://github.com/lxc/lxd/issues/4656
|
Getting the same error with LXC 4.15.
The VM is stopped before I try the delete command. |
|
Having a similar issue with: lxd 5.12-c63881f 24643 |
|
For anyone else experiencing this with LXD 5.x, this might be helpful. It shows what's likely to be where the problem stems from (useful for an lxd dev maybe?), and shows how to fix it when it occurs: #11168 (comment) At least, that's what worked for me. 😄 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Minty fresh Ubuntu 18.04 system
LXD v3.0.0 (latest from apt, how to get v3.0.1?)
Started seeing this beginning last week crop up arbitrarily across my infrastructure. Out of ~10 delete operations, I have seen this happen to 3 containers on 2 different systems.
Tried googling around a bit and I have tried the most common tips on figuring out what might be keeping the dataset busy: There are no snapshots or dependencies, dataset is unmounted i.e.
zfs listreportsCould LXD still be holding the dataset? I see there are a number of zfs related fixes in v3.0.1 but I cannot do an apt upgrade to this version..?
Edit: issuing
systemctl restart lxddoes not resolve the issue, so maybe not lxd after all. Strange...The text was updated successfully, but these errors were encountered: