Network error when using a NFS volume with a container

[FractalWire]

Description

I have a two server setup, one with a NFS server on it, my local machine, and another in Digital Ocean. Both servers are connected via a VPN.

I'm trying to use a NFS volume to keep a persistent postgres data.

If I start a postgres container with that volume on the local machine with my NFS server, everything works fine.

If I start a postgres container on the remote server without the NFS volume, I can connect to it via psql command-line.

Command used to start my container without a NFS volume

$ docker container run -dt --rm -p 5432:5432 -e 'POSTGRES_PASSWORD=postgres' -e 'PGDATA=/var/lib/postgresql/data/pgdata' --name=postgres postgres:9.6

If I start a postgres container on the remote server with the NFS volume I can't connect to it via psql command-line :

$ psql -h 10.0.110.3 postgres postgres

psql: n'a pas pu se connecter au serveur : Connexion refusée

	Le serveur est-il actif sur l'hôte « 10.0.110.3 » et accepte-t-il les connexions
	TCP/IP sur le port 5432 ?

Command used to start my container with a NFS volume

$ docker container run -dt --rm -p 5432:5432 --mount src=test-pgsql,target=/var/lib/postgresql/data/pgdata -e 'POSTGRES_PASSWORD=postgres' -e 'PGDATA=/var/lib/postgresql/data/pgdata' --name=postgres postgres:9.6

volume test-pgsql

$ docker volume inspect test-pgsql 
[
    {
        "CreatedAt": "2019-08-05T10:21:01Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/test-pgsql/_data",
        "Name": "test-pgsql",
        "Options": {
            "device": ":/srv/nfs/docker/pgdata",
            "o": "addr=10.0.110.2",
            "type": "nfs"
        },
        "Scope": "local"
    }
]

Doing a netstat on the remote server, it seems the port is open :

# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
[...]
tcp6       0      0 :::5432                 :::*                    LISTEN      1536/docker-proxy
[...]

But a nmap tells me otherwise :

$ nmap 10.0.110.3 -p 5432

Host is up (0.069s latency).

PORT     STATE  SERVICE
5432/tcp closed postgresql

Describe the results you received:

Impossible to reach the service when using a NFS volume

Describe the results you expected:

Being able to reach the service with or without a NFS volume

Output of docker version:

docker version manager node :

Client:
 Version:           19.03.1-ce
 API version:       1.40
 Go version:        go1.12.7
 Git commit:        74b1e89e8a
 Built:             Sat Jul 27 21:08:50 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.1-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.7
  Git commit:       74b1e89e8a
  Built:            Sat Jul 27 21:08:28 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.7.m
  GitCommit:        85f6aa58b8a3170aec9824568f7a31832878b603.m
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

docker version worker node :

Client: Docker Engine - Community
 Version:           19.03.1
 API version:       1.40
 Go version:        go1.12.5
 Git commit:        74b1e89
 Built:             Thu Jul 25 21:21:05 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.1
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.5
  Git commit:       74b1e89
  Built:            Thu Jul 25 21:19:41 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.6
  GitCommit:        894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc:
  Version:          1.0.0-rc8
  GitCommit:        425e105d5a03fabd737a126ad93d62a9eeede87f
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

docker info manager node :

Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 83
 Server Version: 19.03.1-ce
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: rl5uwqk9kosrqpljht6st2fud
  Is Manager: true
  ClusterID: a90g4285z47e8xgxyp4xar00l
  Managers: 1
  Nodes: 2
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.0.110.2
  Manager Addresses:
   10.0.110.2:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 85f6aa58b8a3170aec9824568f7a31832878b603.m
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.2.4-arch1-1-ARCH
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 5.664GiB
 Name: Stomatopoda
 ID: HZOY:SCC2:7ZHN:BLIK:SPHU:F5OR:R6WO:HZA2:JAVZ:65LI:AMIW:WMK7
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: fractalwire
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

docker info worker node :

Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 19.03.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: q70emxzk01cztp42kix10zq06
  Is Manager: false
  Node Address: 10.0.110.3
  Manager Addresses:
   10.0.110.2:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-55-generic
 Operating System: Ubuntu 18.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 985.5MiB
 Name: docker-node
 ID: 355W:PT7U:SDTV:XCRG:YUC3:DZVS:BJIP:33XV:FV2A:J4YT:F5JR:7U33
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

Working with my local machine and a server on digital ocean connected via a VPN.