newly handed out docker volume mounts can refer to deleted dots

[lukemarsden]

/var/lib/dotmesh/container_mnt/admin # ls -alh
lrwxrwxrwx    1 root     root          62 Jul  5 13:06 dotscience-project -> /var/lib/dotmesh/mnt/dmfs/397bd76b-34d9-4b64-5248-13e850172215

BUT

luke@dot:~/ps/dotscience-ui-spike$ dm dot show dotscience-project
Dot admin/dotscience-project:
Master branch ID: caa5f787-7271-4f01-62d8-d4dca6b2fff8
Dot is currently selected.
Commits: 1
Dot size: 19.00 kiB (all clean)
Branches:
* master
   Replication Status:
    server c1259a93e0498e4d [MASTER] (status: active: waiting, 1 snaps (v0)) is up to date

Note that the fsIds do not line up. This happened after I deleted a dot and tried to make a new one with the same name. I guess the symlink was still lying around. Isn't deletion meant to clean up these /var/lib/dotmesh/container_mnt symlinks?

[lukemarsden]

manually fixed with

/var/lib/dotmesh/container_mnt/admin # unlink dotscience-project

and then clicking the "docker rm -f all the containers" button on the dotscience ui, which results in them being recreated.

and so, a valid symlink was formed:

lrwxrwxrwx    1 root     root          62 Jul  5 14:24 dotscience-project -> /var/lib/dotmesh/mnt/dmfs/caa5f787-7271-4f01-62d8-d4dca6b2fff8

[lukemarsden]

trying this again didn't exhibit the same problem, so it's at least intermittent fwiw.

[lukemarsden]

this happened after #488 fwiw

[alaric-dotmesh]

There's some code already in place to delete the symlinks, but it's being missed or coming too late in some error cases, so I've added code to do a synchronous deletion on the current node when Delete happens - which should create a somewhat better illusion of the delete happening synchronously (from one vantage point).