Cannot remove NFS volumes created by podman volume create

[toddhpoole]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Based on our reading of the 1.6.0 changelog, podman volume create can now "create and mount volumes with options, allowing volumes backed by NFS." When we try to exercise this feature and create an NFS-backed volume, we're then unable to remove it.

Documentation covering NFS volumes is non-existent, so if this is user error, please advise. Either way, expanding the Examples section of podman-volume-create.1.md and podman-volume-rm.1.md with more examples, including several NFS ones, would be helpful.

Steps to reproduce the issue:

1. Create a volume backed by an NFS filesystem (guessing at the invocation here... again, there are no NFS examples in the documentation to reference).

$ podman volume create --opt type=nfs --opt o=addr=192.168.2.126,rw --opt device=:/exports/test test_vol_name
test_vol_name

2. Confirm that podman is aware of the volume.

$ podman volume inspect --all
[
     {
          "Name": "test_vol_name",
          "Driver": "local",
          "Mountpoint": "/home/testuser/.local/share/containers/storage/volumes/test_vol_name/_data",
          "CreatedAt": "2019-10-12T19:02:47.148481423-07:00",
          "Labels": {
               
          },
          "Scope": "local",
          "Options": {
               
          }
     }
]

3. Try to remove the volume:

$ podman volume rm test_vol_name
Error: error unmounting volume test_vol_name: error unmounting volume test_vol_name: invalid argument

Describe the results you received:
Unable to remove the volume; received following error message:
Error: error unmounting volume test_vol_name: error unmounting volume test_vol_name: invalid argument

Describe the results you expected:
Volume removed; no error message received.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:            1.6.1
RemoteAPI Version:  1
Go Version:         go1.12.9
OS/Arch:            linux/amd64

Output of podman info --debug:

$ podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.9
  podman version: 1.6.1
host:
  BuildahVersion: 1.11.2
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.1-1.fc30.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.1, commit: 4346fbe0b2634b05857973bdf663598081240374'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 31369199616
  MemTotal: 33539690496
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 16840126464
  SwapTotal: 16840126464
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: host0
  kernel: 5.2.18-200.fc30.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-4.git19d199a.fc30.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.2
      commit: 19d199a6ca424fcf9516320a327cedad85cf4dfb
  uptime: 3h 28m 18.42s (Approximately 0.12 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/testuser/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.6.3-2.0.dev.git46c0f8e.fc30.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.6.3
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/testuser/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /run/user/1000
  VolumePath: /home/testuser/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-1.6.1-2.fc30.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Fresh minimal install of Fedora 30 with yum -y install vim nfs-utils podman buildah.

Exports are visible to host:

$ showmount -e 192.168.2.126
Export list for 192.168.2.126:
/exports/test  192.168.2.0/24

Exports can be mounted outside of podman using mount 192.168.2.126:/exports/test /mnt/test.

[rhatdan]

This looks like a bug.
@mheon PTAL

[mheon]

The volume does not appear to actually be mounting which is why we get an EINVAL. Easy solution is to catch that and ignore it, but handing real unmount errors in a way that still allows the volume to be removed needs to happen as well.

…

On Sun, Oct 13, 2019, 07:02 Daniel J Walsh ***@***.***> wrote: This looks like a bug. @mheon <https://github.com/mheon> PTAL — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4247>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCH7BXHV7W3JXC52U7TQOL543ANCNFSM4JAFIHGA> .

[mheon]

#4256 to resolve this and #4248