docker cp to attached volume in container causes an unmount to be issued #93
Comments
|
After further investigation this appears to be a problem with Docker and some assumptions that may be made regarding iSCSI connections. There are different ways to creat an iSCSI target for a device, one is to use unique target IQN's for each "volume" that exists on a device. This is how SolidFire works. The other is to used a shared TGT IQN and access individual volumes via iscsi portal/lun assignment. A number of devices use this method including the ONTAP devices. The This won't work for devices that use individual targets and could potentially cause issues for others. To work around this the best option is to mount the data source in to the container as well. From there use the container FS to copy the data from the source to the attached NDVP volume in the container. So something like this: |
|
@j-griffith Eeww! Is there an issue open with the Moby project? I searched but couldn't find one. |
|
I have a fix using ref counting here, there's definitely some things we should've been doing better here. |
SolidFire is a bit different than some other devices in the portfolio in that it uses a unique target for each volume and it doesn't do any internal reference counting in terms of attachment counts. This gets pretty wonky in Docker becuase docker does a lot of mount/unmount calls in the background, particularly when using commands like `docker cp ./some-data container-foo:/data/` It's debatable whether docker is well behaved here, but regardless we can and should do better in terms of handling things like this. This change adds a simple mechanism where we count the number of Mounts issued against a volume. We increment this count each time a new mount request is received, and decrement it each time an unmount request is received. This way we don't accidentally rip a volume out from under a container during things like docker cp and more importantly we don't blow up if somebody is trying to do multiple attachments to a single volume from multiple containers on the same node. For now this isn't persistent, and it can also cause some difficulty with hanging connections if the counter somehow gets out of sync. Those types of things should be considered/addressed in the future, and we will probably need somebody to get on creating an admin cli like the solidfire-docker-driver has to do clean ups etc. Addresses Github Issue: NetApp#93
|
Preview of fix here: https://github.com/j-griffith/netappdvp/tree/refcount_attachments_in_solidfire |
There are soem questionable behaviors on the docker side wrt sequences of mounting and unmounting. The `docker cp` command inparticular has a good bit of seemingly unnecessary mount/unmount operations that for some devices (those that use unique targets) can trip things up a bit. The problems include removing a device that's in use by a container as well as removing an attachment that is expected to be present. One way of dealing with this (and it's called out in the Docker Volume API docs) is to utilize reference counting of the mounts and track things that way. We could do that, but there's some concern around that in terms of things getting out of sync and also storing state in the driver isn't the most ideal thing either. It turns out however that maybe none of this is really necessary. Instead of always unmounting nad removing iSCSI connections we can instead leave mounts and iscsi connections in place for the life of a volume. So what this change does is makes the Docker API Unmount basically a NOOP. We still do things like Mount and create iSCSI connection the same as before, but now we ignore the Unmount command. If subsequent or multiple Mount requests come in for a volume that's fine, we just reuse the connection and mountpoint that we already have in place. We leave this in place until the volume is removed. This works will in the case of things like reboots, or restarts of NDVP as well, because we don't need to worry about state or ref counting. If a Mount cmd is received in those scenarios and the mount was lost due to a reboot, we just create one and go about our business. Addresses Github Issue: NetApp#93
There are some questionable behaviors on the docker side wrt sequences of mounting and unmounting. The `docker cp` command in particular has a good bit of seemingly unnecessary mount/unmount operations that for some devices (those that use unique targets) can trip things up a bit. The problems include removing a device that's in use by a container as well as removing an attachment that is expected to be present. One way of dealing with this (and it's called out in the Docker Volume API docs) is to utilize reference counting of the mounts and track things that way. We could do that, but there's some concern around that in terms of things getting out of sync and also storing state in the driver isn't the most ideal thing either. It turns out however that maybe none of this is really necessary. Instead of always unmounting and removing iSCSI connections we can instead leave mounts and iscsi connections in place for the life of a volume. So what this change does is makes the Docker API Unmount basically a NOOP. We still do things like Mount and create iSCSI connection the same as before, but now we ignore the Unmount command. If subsequent or multiple Mount requests come in for a volume that's fine, we just reuse the connection and mount point that we already have in place. We leave this in place until the volume is removed. This works will in the case of things like reboots, or restarts of NDVP as well, because we don't need to worry about state or ref counting. If a Mount cmd is received in those scenarios and the mount was lost due to a reboot, we just create one and go about our business. Addresses Github Issue: NetApp#93
|
The change in 29aceaf should work around the problem by delaying the iSCSI session teardown until we delete the volume. The 'docker cp' commands that failed above work for me after this change. |
If you try and use to xfr data from node to a mounted volume in a container the result is an unMount being sent from Docker after the Mount is detected as already existing.
To reproduce:
docker volume create -d netapp --name=foobardocker run -it --volume-driver netapp --volume foobar:/data --name crumbs bashThen on the host:
docker cp /var/log/syslog- crumbs:/data/syslogAfter running the docker cp command the volume is unmounted by the go-plugin-helper, the logs show this here:
DEBU[2017-08-25T15:21:05Z] Mount(&{foo b995165f99e38f246a0236cda8c2505b5922dc8c0632cc685a28f18ed67f1814}) DEBU[2017-08-25T15:21:05Z] Mounting volume foo on /var/lib/docker-volumes/solidfire/foo DEBU[2017-08-25T15:21:05Z] Begin osutils.GetDFOutput DEBU[2017-08-25T15:21:05Z] /var/lib/docker-volumes/solidfire/foo already mounted, returning existing mount 2017/08/25 15:21:05 Entering go-plugins-helpers unmountPath DEBU[2017-08-25T15:21:05Z] Unmount(&{foo b995165f99e38f246a0236cda8c2505b5922dc8c0632cc685a28f18ed67f1814}) DEBU[2017-08-25T15:21:05Z] SolidfireSANStorageDriver#Detach(foo, /var/lib/docker-volumes/solidfire/foo) DEBU[2017-08-25T15:21:05Z] Begin osutils.Umount: /var/lib/docker-volumes/solidfire/foo DEBU[2017-08-25T15:21:05Z] Response from umount /var/lib/docker-volumes/solidfire/foo:I think this might be an upstream issue with the Docker volume API but not sure yet.
The text was updated successfully, but these errors were encountered: