runtime leaves mounts around after container shutdown

[jodh-intel]

As noted by @grahamwhaley:

$ sudo docker run -ti busybox true
$ mount|grep /run/hyper/shared/pods
/dev/mapper/zesty--vg-root on /run/hyper/shared/pods/a31e028e712b0710cf0373e418729b5fd8df782ab3463d47faab8e582ad89064/a31e028e712b0710cf0373e418729b5fd8df782ab3463d47faab8e582ad89064-44262945da62c2d6-hostname type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/zesty--vg-root on /run/hyper/shared/pods/a31e028e712b0710cf0373e418729b5fd8df782ab3463d47faab8e582ad89064/a31e028e712b0710cf0373e418729b5fd8df782ab3463d47faab8e582ad89064-25d8a0dc5936aa0a-hosts type ext4 (rw,relatime,errors=remount-ro,data=ordered)

The commit that introduced this issue is the virtcontainers re-vendor commit cec1b6f.

$ git log --format="%B" -1 cec1b6f4d8b9bc71887bc687ed39dc2502ccf9e8|grep mount
4c2c9a4 mounts: Fix bug while checking if /dev was bind-mounted
b3da3de mount: Fix tests for bindMountContainerRootfs function
c4a4be4 mount: Gather the entire bind mount API

c4a4be4 looks like a likely suspect but need to dig into that...

[jodh-intel]

The stray mounts in question are /etc bind mounts. An example from config.json (but not the same one as for the container above:

    { 
      "destination": "/etc/resolv.conf",
      "options": [
        "rbind",
        "rprivate"
      ],
      "source": "/var/lib/docker/containers/75e7b69fa65458418ed7dc008c62b22433a122b076e12b234d2609597fc5b844/resolv.conf",
      "type": "bind"
    },
    {
      "destination": "/etc/hostname",
      "options": [
        "rbind",
        "rprivate"
      ],
      "source": "/var/lib/docker/containers/75e7b69fa65458418ed7dc008c62b22433a122b076e12b234d2609597fc5b844/hostname",
      "type": "bind"
    },
    {
      "destination": "/etc/hosts",
      "options": [
        "rbind",
        "rprivate"
      ],
      "source": "/var/lib/docker/containers/75e7b69fa65458418ed7dc008c62b22433a122b076e12b234d2609597fc5b844/hosts",
      "type": "bind"
    },

What's odd is that there isn't also a bind mount left for /etc/resolv.conf (unless the string host is the key ;)

[grahamwhaley]

Thanks for opening this @jodh-intel
For reference, I do sometimes also see a resolv.conf mount left as well - not as often as the other two, but I have seen it (which smells like a race to me then maybe....)

[jodh-intel]

This is fixed in virtcontainers by containers/virtcontainers#567.

Once that lands, we need to revendor virtcontainers into the runtime of course.

[jodh-intel]

Worth noting that running cc-collect-data.sh actually did log the problem:

time="2018-01-16T10:06:10Z" level=warning msg="Could not umount" error="<nil>" host-path=/run/hyper/shared/pods/abfea09350acb235cc0ac1e55404194aeb092ad75b0b295107270bd58795a205/abfea09350acb235cc0ac1e55404194aeb092ad75b0b295107270bd58795a205-7521959b6352a12a-resolv.conf source=virtcontainers subsystem=mount 

The clue being that error=<nil>.

[jodh-intel]

Blocked on containers/virtcontainers#567.