NFS Volumes not working because NFS server address is not being used properly during underlying mount.nfs command

[zaphodbb]

Bug description
NFS and NFSv4 mounts are not working in Portainer 2.18.4 on Fedora 37 with podman 4.5.0 and nfs-utils 2.6.2. Creating an NFS volume results in a volume with device=":/data" and options="addr=192.168.YY.XX,rw,noatime,rsize=8192,wsize=8192,tcp,timeo=14,nfsvers=4" and type="nfs".

When trying to use this volume on a container, I'm seeing "mount.nfs: Failed to resolve server : Name or service not known" in the logs and the container creation fails.

As I understand it, this ends up executing the command 'mount.nfs -o "addr=192.168.86.81,rw,noatime,rsize=8192,wsize=8192,tcp,timeo=14,nfsvers=4" :/data /var/lib/containers/storage/volumes/Media/_data' whic, run manually, as root on the podman host fails with the exact same error as I'd expect because proper syntax of this command should be 'mount.nfs -o "addr=192.168.YY.XX,rw,noatime,rsize=8192,wsize=8192,tcp,timeo=14,nfsvers=4" 192.168.YY.XX:/data /var/lib/containers/storage/volumes/Media/_data'

mount.nfs will not mount a device named :/<export path>, it should be <nfs server>:/<export path>. IE: device=":/data" needs to be device="192.168.YY.XY:/data". for mount.nfs, '-o addr=192.168.YY.XX' along with a device of ':/<export path>' will not work becuase that's not the right syntax for mount.nfs.

Expected behavior
Expectation is that the NFS export "/data" on server 192.168.YY.XX would be mounted on /var/lib/containers/storage/volumes/Media/_data and attached to the container properly

Portainer Logs
Jul 22 16:24:45 docker-apps podman[233717]: Copying blob sha256:6f7407fa46eab690743e33af5ad9ee662f2ccda0a1052cff96ca58e47705938a
Jul 22 16:24:45 docker-apps podman[233717]: Copying blob sha256:22f95cb725a65d47d874443fb62cec705744ccf1faccf877dbd72345a5e28844
Jul 22 16:24:45 docker-apps podman[233717]: Copying blob sha256:b356d64974d89f3a5e652b921212eb40b83e3f83b1f8211faf46eedacf6c8019
Jul 22 16:24:45 docker-apps podman[233717]: Copying blob sha256:540082806dea3da082efe21548361c38b78b945c302b9fbcb93b8adbc8c250df
Jul 22 16:24:45 docker-apps podman[233717]: Copying blob sha256:fef37c725dc213ede4d8688ebac9a6c1b23aca45c26a073b5a621a12eec18b15
Jul 22 16:24:45 docker-apps podman[233717]: Copying blob sha256:8947dc7e7c3f61292b6a11e5fd280f0400fb362c9f506c6df49bbd10e66b6952
Jul 22 16:24:45 docker-apps podman[233717]: Copying blob sha256:bd58523eafd4b71ead63f526e1ac7f210a0cc69bae7f0477573085705634331c
Jul 22 16:24:45 docker-apps podman[233717]: Copying blob sha256:e6120b250abf28eb9120db3d39496f2ceacb480c6ba76a05e175195e05e49741
Jul 22 16:24:45 docker-apps podman[233717]: Copying config sha256:a64d9a9c03e5d38e0c4f5827419177e19e755addfe8450b788b6587d5835304b
Jul 22 16:24:45 docker-apps podman[233717]: Writing manifest to image destination
Jul 22 16:24:45 docker-apps podman[233717]: Storing signatures
Jul 22 16:24:45 docker-apps podman[233717]: @ - - [22/Jul/2023:16:24:44 -0400] "POST /images/create?fromImage=binhex%2Farch-plexpass:latest HTTP/1.1" 200 824 "https://docker-apps:9443/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"
Jul 22 16:24:45 docker-apps podman[233717]: 2023-07-22 16:24:45.197636822 -0400 EDT m=+6.590785234 image pull a64d9a9c03e5d38e0c4f5827419177e19e755addfe8450b788b6587d5835304b docker.io/binhex/arch-plexpass:latest
Jul 22 16:24:45 docker-apps podman[233717]: 2023-07-22 16:24:45.250099702 -0400 EDT m=+6.643248115 container create 0100435669c4f37193f550b497b302305346e78f38e7b8e578c4e529d6803394 (image=docker.io/binhex/arch-plexpass:latest, name=PlexPass, org.opencontainers.image.source== https://github.com/binhex/arch-plexpass, org.opencontainers.image.authors== binhex)
Jul 22 16:24:45 docker-apps podman[233717]: @ - - [22/Jul/2023:16:24:45 -0400] "POST /containers/create?name=PlexPass HTTP/1.1" 201 88 "https://docker-apps:9443/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"
Jul 22 16:24:45 docker-apps podman[233787]: [INFO netavark::firewall] Using iptables firewall driver
Jul 22 16:24:45 docker-apps systemd[1]: var-lib-containers-storage-overlay-f944e9f5f8ea9b78d182d8b65a0a76518e50183cacf1658fb2674805fef27288-merged.mount: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit var-lib-containers-storage-overlay-f944e9f5f8ea9b78d182d8b65a0a76518e50183cacf1658fb2674805fef27288-merged.mount has successfully entered the 'dead' state.
Jul 22 16:24:45 docker-apps systemd[1]: var-lib-containers-storage-overlay\x2dcontainers-0100435669c4f37193f550b497b302305346e78f38e7b8e578c4e529d6803394-userdata-shm.mount: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit var-lib-containers-storage-overlay\x2dcontainers-0100435669c4f37193f550b497b302305346e78f38e7b8e578c4e529d6803394-userdata-shm.mount has successfully entered the 'dead' state.
Jul 22 16:24:45 docker-apps systemd-networkd[94]: veth4: Link UP
Jul 22 16:24:45 docker-apps systemd-networkd[94]: veth4: Gained carrier
Jul 22 16:24:45 docker-apps podman[233787]: [INFO netavark::network::netlink] Adding route (dest: 0.0.0.0/0 ,gw: 10.88.0.1, metric 100)
Jul 22 16:24:45 docker-apps podman[233813]: [INFO netavark::firewall] Using iptables firewall driver
Jul 22 16:24:45 docker-apps systemd-networkd[94]: veth4: Link DOWN
Jul 22 16:24:45 docker-apps systemd-networkd[94]: veth4: Lost carrier
Jul 22 16:24:45 docker-apps systemd[1]: run-netns-netns\x2d385a774e\x2d2136\x2d49ae\x2d08b2\x2d9713b71f6f15.mount: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit run-netns-netns\x2d385a774e\x2d2136\x2d49ae\x2d08b2\x2d9713b71f6f15.mount has successfully entered the 'dead' state.
Jul 22 16:24:45 docker-apps podman[233717]: time="2023-07-22T16:24:45-04:00" level=info msg="Request Failed(Internal Server Error): mounting volume Media for container 0100435669c4f37193f550b497b302305346e78f38e7b8e578c4e529d6803394: mount.nfs: Failed to resolve server : Name or service not known\n"
Jul 22 16:24:45 docker-apps podman[233717]: @ - - [22/Jul/2023:16:24:45 -0400] "POST /containers/0100435669c4f37193f550b497b302305346e78f38e7b8e578c4e529d6803394/start HTTP/1.1" 500 273 "https://docker-apps:9443/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"
Jul 22 16:24:45 docker-apps podman[233717]: @ - - [22/Jul/2023:16:24:45 -0400] "GET /containers/json?all=1&filters=%7B%22name%22:%5B%22%5E%2FPlexPass$%22%5D%7D HTTP/1.1" 200 1341 "https://docker-apps:9443/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"
Jul 22 16:24:45 docker-apps podman[233717]: 2023-07-22 16:24:45.46259804 -0400 EDT m=+6.855746453 container remove 0100435669c4f37193f550b497b302305346e78f38e7b8e578c4e529d6803394 (image=docker.io/binhex/arch-plexpass:latest, name=PlexPass, org.opencontainers.image.authors== binhex, org.opencontainers.image.source== https://github.com/binhex/arch-plexpass)
Jul 22 16:24:45 docker-apps podman[233717]: @ - - [22/Jul/2023:16:24:45 -0400] "DELETE /containers/0100435669c4f37193f550b497b302305346e78f38e7b8e578c4e529d6803394?v=1&force=true HTTP/1.1" 204 0 "https://docker-apps:9443/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0"

Steps to reproduce the issue:

1. Access portainer GUI
2. Create a new volume named "Media", driver=local
3. Select "Use NFS volume"
4. under 'NFS Settings': Address = IP address of NFS server, NFS Version = either NFS or NFS4 (problem exists with either), mount point = ":/data" or "/data" (same issue either way), options = "rw,noatime,rsize=8192,wsize=8192,tcp,timeo=14" (default values)
5. Click "Create the volume"
6. Volume creates with no issues
7. On the podman host, as root, execute "journalctl -flax", allowing logs to tail while you create the container to leverage the volume that you created
8. Create a Container leveraging this volume and hit "deploy the container"
9. Results in a "500 server error"
10. On the podman host, review logs with journalctl and find "mount.nfs: Failed to resolve server : Name or service not known"
11. On podman host, as root, execute "podman volume inspect Media" and notice that the device value under Options starts with ":" which is invalid for the mount.nfs command.

Technical details:

• Portainer version: 2.18.4
• Podman version (managed by Portainer): 4.5.0
• Kubernetes version (managed by Portainer): N/A
• Platform (windows/linux): Fedora 37
• Command used to start Portainer (docker run -p 9443:9443 portainer/portainer): sudo podman run -d -p 9443:9443 --privileged -v /nfs/podmanappcfg/portainer:/data -v /run/podman/podman.sock:/var/run/docker.sock:Z --name portainer-ce portainer/portainer-ce
• Browser: Firefox 115.0.2
• Use Case (delete as appropriate): Using Portainer at Home
• Have you reviewed our technical documentation and knowledge base? Yes

Additional context
In portainer issue 4303 (containers/podman#4303), mheon notes that "I will note that Podman is invoking the mount binary using the arguments you provide - mount -t -o (where type is from --opt type=..., o from --opt o=, and device is from --opt device=...."

Running this exact mount command with the device, options, and type provided from "podman volume inspect Media", I get the exact same error. Simply placing the NFS Server name or address on the front of the device value (thus getting <address>:/<export path>), the mount command executes without error.

[zaphodbb]

Sorry, in description paragraph 4, "mount.nfs will not mount a device named :/, it should be :/." is not complete, that should have read "mount.nfs will not mount a device named :/<export path>, it should be ip address or hostname:/<export path>" This is simply stating that in mount.nfs, the device name cannot start with ":", it has to be a hostname or address and the export path separated by a colon.

device: ":/data" is invalid
device: "192.168.0.12:/data" is valid
device: "storage.mydomain.com:/data" is valid as well.

Apparently the markdown doesn't like less than/greater than symbols.

[zaphodbb]

Issue appears to be in app/docker/views/volumes/create/createVolumeController.js within the prepareNFSConfiguration function on lines 91 to 104. Specifically, lines 102 and 103 prepend a ":" to the "mountPoint" value (102) which then becomes the device value (103). There should be a line between 102 and 103 that sets mountPoint to data.serverAddress + mountPoint which would result in the proper <serverAddress>:<mountPoint>

      var mountPoint = data.mountPoint[0] === ':' ? data.mountPoint : ':' + data.mountPoint;
      mountPoint = data.serverAddress + mountPoint; 
      driverOptions.push({ name: 'device', value: mountPoint });

[zaphodbb]

I was able to confirm that if I execute podman volume create --driver local --opt type=nfs --opt device=192.168.XX.YY:/data --opt o="rw,noatime,rsize=8192,wsize=8192,tcp,timeo=14,nfsvers=4" Media, a volume is created that has the device=<ip address>:<export> format and using this volume in my container works as expected. So, for now it seems that the workaround is to create NFS volumes using the podman CLI and then leveraging these volumes in portainer containers (the new volume showed up in portainer after it was created in the podman CLI and the volumes page in portainer was reloaded)

[jamescarppe]

I suspect this might be a podman (or Fedora) specific issue, as I've just followed your steps on a Ubuntu Docker host and was able to attach my NFS volume to a container without issue.

Note we do not currently support podman with Portainer - while some functionality may work, there are certainly issues (we document some of them here).

[zaphodbb]

While I'll say that's fair and I'll continue to use my workaround ... I'll still say that the "device" value portainer creates for an nfs volume is wrong. An NFS volume should be <ip address>:<export path>, not :<export path>, that's a simple fact. The fact that this works in ubuntu/docker indicates that either docker or ubuntu's mount.nfs is munging things and using the addr= option to decide what IP to use for the mount ... neither of these are actually standard practices for NFS.

Even Ubuntu's official NFS documentation (https://ubuntu.com/server/docs/service-nfs) uses <ip address>:<export path> and making this change would allow increased compatibility with podman and/or RPM based distros (and >should not< break compatibility with docker/Ubuntu). NFS device should always start with IP address of the NFS server for NFS mount points.

[jamescarppe]

Since we were initially an interface for Docker we followed the Docker standards when doing our implementation. The Docker documentation includes an example where a volume is created alongside a service and the device value there is specified as :/var/docker-nfs, without the address. I'm not sure if there's an existing issue in the Docker repo around fixing the way they do this.

In saying that, we should probably handle at least the specification of the mount point a bit better in the UI, at the very least allowing you to specify an address alongside the mount point (192.168.1.1:/volume/nfs) to handle cases like this. I'll have a chat with the team.

[zaphodbb]

Thank you James, I appreciate the effort regardless of the outcome. Here's hoping that a simple resolution can increase compatibility.

[cgoudie]

Centos EL9, definitely a problem. Definitely agree with @jamescarppe

[soupdiver]

I'm running into the same problem

[StinkyTACO]

how are we months later with no follow up and this still being a problem ?

i ended up here hoping i might be an idiot not knowing what im doing but no a months long issue with no haste to fix it seems.