FreeBSD 12.0 jails dying

[denisSal]

In FreeBSD 12.0, sometimes jails cannot be destroyed and removed from the existing jail list, but rather stay in the "dying" state (this can be seen with jls -ad). This shouldn't be a problem except when trying to restart the experiment, when jails are destroyed and created with the same ID.

We could bypass this issue by matching the current experiment ID with the running jail IDs before running every experiment and changing it if it already exists.

[Firerouge]

The problem of repeating jail IDs could be addressed by using random jail IDs in collisions, however; my use case requires repeatedly using the same experiment ID.
To fix one without breaking the other, the change would require supporting cases where experiment and jail ID are no longer the same.

The underlying problem is FreeBSD 12 (stable) doesn't complete deleting jails upon experiment termination.

[denisSal]

I've been digging a bit on my FreeBSD 12.0 virtual machine so I just wanted to write an update on this issue...

First of all, this is definitely primarily a bug with FreeBSD jails (even without IMUNES). I can replicate it with:

# jail -c name=foo persist
# jexec foo sh
# jail -r foo

There shouldn't be any, but there are dying jails left when I run jls -d. This bug report also shows this.

I accidentally found out that the dying jails disappear when copying/creating a big file on my VM, for example:

# dd if=/dev/zero of=/tmp/bigfile bs=1M count=500

This also works with IMUNES dying jails. Of course, this isn't a fix or anything, I just wanted to acknowledge this 'hack' (because this and this tells us that there is no fix for this, only that 'keeping the same JID or ... recycling freed JID is not recommended' and to just 'ignore it'. :)

This and again this suggest that there is a problem with credentials holding a jail.

To fix one without breaking the other, the change would require supporting cases where experiment and jail ID are no longer the same.

This seems like a correct solution to the IMUNES issue. I will look into this but it could be a bigger challenge for me to test and make sure I don't break anything so it could take a while.

[denisSal]

This patch resolves the issue, and jails now don't remain in the dying state:
patch.zip

WARNING: I am not a FreeBSD developer and this is not an official patch, but according to this it should be ok to revert this one line to an older revision.

How to patch:

# cd /tmp/
# unzip patch.zip
# cd /usr/src
# patch < /tmp/vfs_bio.patch
# cd sys/amd64/conf # or i386
# config GENERIC
# cd ../compile/GENERIC
# make -j4 depend && make -j4
# make -j4 install
# reboot