adosztal/net-toolbox docker error on GNS server 2.2

[32e3e32]

Hi all,
First a big thank you to everyone contributing to GNS3!!!

Now to the problem: I'm getting an error when adding docker device to topology. GNS Server 2.2.0dev12 running on Ubuntu 18. Later reinstalled Ubuntu and updated server and client to 2.2.0b2 but problem with "duplicate persistent volumes" is also "persisting"...

FYI, Other docker containers like webterm are working fine.

Here's the log:
`
2019-06-03 02:36:50 INFO web_log.py:233 192.168.0.211 [03/Jun/2019:02:36:50 +0000] "DELETE /v2/templates/51ca3751-6d1c-4069-af49-ed9e758f95d8 HTTP/1.1" 204 213 "-" "GNS3 QT Client v2.2.0dev12"
2019-06-03 02:37:16 INFO notification_handler.py:82 Client has disconnected from controller WebSocket
2019-06-03 02:37:16 INFO project_handler.py:279 Client has disconnected from notification stream for project ID '2040f60a-7ffa-4263-b3df-ae126169ce27' (WebSocket method)
2019-06-03 02:37:21 INFO project_handler.py:288 Project '2040f60a-7ffa-4263-b3df-ae126169ce27' is automatically closing due to no client listening
2019-06-03 02:37:21 INFO web_log.py:233 127.0.0.1 [03/Jun/2019:02:37:21 +0000] "POST /v2/compute/projects/2040f60a-7ffa-4263-b3df-ae126169ce27/ethernet_switch/nodes/1158f330-e9f8-43c5-9f3a-8a573fa2943a/stop HTTP/1.1" 204 256 "-" "Python/3.6 aiohttp/3.5.4"
2019-06-03 02:37:21 INFO ethernet_switch.py:248 Ethernet switch "Ethernetswitch-1" [1158f330-e9f8-43c5-9f3a-8a573fa2943a] has been deleted
2019-06-03 02:37:21 INFO hypervisor.py:148 Stopping Dynamips process PID=2399
2019-06-03 02:37:21 INFO project.py:361 Project 2040f60a-7ffa-4263-b3df-ae126169ce27 with path '/home/gns3/GNS3/projects/2040f60a-7ffa-4263-b3df-ae126169ce27' closed
2019-06-03 02:37:21 WARNING project.py:364 Project 2040f60a-7ffa-4263-b3df-ae126169ce27 has TCP ports still in use: {5001, 5002}
2019-06-03 02:37:21 INFO web_log.py:233 127.0.0.1 [03/Jun/2019:02:37:21 +0000] "POST /v2/compute/projects/2040f60a-7ffa-4263-b3df-ae126169ce27/close HTTP/1.1" 204 225 "-" "Python/3.6 aiohttp/3.5.4"

---------- updated client to latest 2.2.0b2 ---------------

2019-06-03 02:39:59 INFO web_log.py:233 192.168.0.211 [03/Jun/2019:02:39:59 +0000] "GET /v2/version HTTP/1.1" 200 230 "-" "GNS3 QT Client v2.2.0b2"
2019-06-03 02:39:59 INFO notification_handler.py:73 New client has connected to controller WebSocket

2019-06-03 02:40:11 INFO project.py:78 Project 2040f60a-7ffa-4263-b3df-ae126169ce27 with path '/home/gns3/GNS3/projects/2040f60a-7ffa-4263-b3df-ae126169ce27' created
2019-06-03 02:40:11 INFO web_log.py:233 127.0.0.1 [03/Jun/2019:02:40:11 +0000] "POST /v2/compute/projects HTTP/1.1" 201 299 "-" "Python/3.6 aiohttp/3.5.4"
2019-06-03 02:40:11 INFO init.py:303 Creating new hypervisor 0.0.0.0:37329 with working directory /home/gns3/GNS3/projects/2040f60a-7ffa-4263-b3df-ae126169ce27/project-files/dynamips
2019-06-03 02:40:11 INFO hypervisor.py:127 Starting Dynamips: ['/usr/local/bin/dynamips', '-N1', '-l', 'dynamips_i2_log.txt', '-H', '37329']
2019-06-03 02:40:11 INFO hypervisor.py:129 Dynamips process logging to /home/gns3/GNS3/projects/2040f60a-7ffa-4263-b3df-ae126169ce27/project-files/dynamips/dynamips_i2_stdout.txt
2019-06-03 02:40:11 INFO hypervisor.py:136 Dynamips process started PID=2691
2019-06-03 02:40:11 INFO init.py:305 Hypervisor 0.0.0.0:37329 has successfully started
2019-06-03 02:40:11 INFO dynamips_hypervisor.py:92 Connected to Dynamips hypervisor on 127.0.0.1:37329 after 0.4491 seconds
2019-06-03 02:40:11 INFO ethernet_switch.py:183 Ethernet switch "Ethernetswitch-1" [1158f330-e9f8-43c5-9f3a-8a573fa2943a] has been created
2019-06-03 02:40:11 INFO web_log.py:233 127.0.0.1 [03/Jun/2019:02:40:11 +0000] "POST /v2/compute/projects/2040f60a-7ffa-4263-b3df-ae126169ce27/ethernet_switch/nodes HTTP/1.1" 201 1571 "-" "Python/3.6 aiohttp/3.5.4"
2019-06-03 02:40:11 INFO web_log.py:233 192.168.0.211 [03/Jun/2019:02:40:11 +0000] "POST /v2/projects/2040f60a-7ffa-4263-b3df-ae126169ce27/open HTTP/1.1" 201 764 "-" "GNS3 QT Client v2.2.0b2"
2019-06-03 02:40:11 INFO project_handler.py:270 New client has connected to the notification stream for project ID '2040f60a-7ffa-4263-b3df-ae126169ce27' (WebSocket method)

2019-06-03 02:40:27 WARNING symbols.py:135 Could not retrieve symbol 'fortinet.svg'
2019-06-03 02:40:27 WARNING symbols.py:135 Could not retrieve symbol 'juniper-vmx.svg'
2019-06-03 02:40:27 WARNING symbols.py:135 Could not retrieve symbol 'parrotlogo.png'
2019-06-03 02:40:27 WARNING symbols.py:135 Could not retrieve symbol 'linux_guest.svg'
2019-06-03 02:40:27 WARNING symbols.py:135 Could not retrieve symbol 'microsoft.svg'
2019-06-03 02:40:27 WARNING symbols.py:135 Could not retrieve symbol 'loadbalancer.svg'

2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'juniper-vqfx.svg'
2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'securityonion-logo.png'

2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'mgmt_station.svg'
2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'ostinato-3d-icon.svg'
2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'cisco-fmcv.svg'
2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'cisco-ise.svg'
2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'pan-vm-fw.svg'
2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'firefox.svg'
2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'mgmt_station_docker.svg'
2019-06-03 02:40:28 WARNING symbols.py:135 Could not retrieve symbol 'vRIN.svg'

2019-06-03 02:40:57 INFO docker_vm.py:1035 Docker container "Toolbox-1" [95b8098b-c45a-440a-9011-40173d41f8da]: number of Ethernet adapters changed to 1
2019-06-03 02:40:57 INFO docker_vm.py:317 Image 'adosztal/net_toolbox:latest' is missing, pulling it from Docker hub...

2019-06-03 02:41:26 INFO web_log.py:233 192.168.0.211 [03/Jun/2019:02:41:26 +0000] "GET /v2/computes HTTP/1.1" 200 1024 "-" "GNS3 QT Client v2.2.0b2"

------------------- Here's the main error -----------------------------
2019-06-03 02:41:28 ERROR route.py:211 Node error detected: DockerError
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/gns3_server-2.2.0b2-py3.6.egg/gns3server/web/route.py", line 196, in control_schema
await func(request, response)
File "/usr/local/lib/python3.6/dist-packages/gns3_server-2.2.0b2-py3.6.egg/gns3server/handlers/api/compute/docker_handler.py", line 65, in create
extra_volumes=request.json.get("extra_volumes"))
File "/usr/local/lib/python3.6/dist-packages/gns3_server-2.2.0b2-py3.6.egg/gns3server/compute/base_manager.py", line 259, in create_node
await node.create()
File "/usr/local/lib/python3.6/dist-packages/gns3_server-2.2.0b2-py3.6.egg/gns3server/compute/docker/docker_vm.py", line 335, in create
"Binds": self._mount_binds(image_infos),
File "/usr/local/lib/python3.6/dist-packages/gns3_server-2.2.0b2-py3.6.egg/gns3server/compute/docker/docker_vm.py", line 267, in _mount_binds
raise DockerError("Duplicate persistent volume {} detected.\n\nVolumes specified in docker image as well as user specified persistent volumes must be unique.".format(volume))
gns3server.compute.docker.docker_error.DockerError: Duplicate persistent volume /etc detected.

Volumes specified in docker image as well as user specified persistent volumes must be unique.
2019-06-03 02:41:28 INFO web_log.py:233 127.0.0.1 [03/Jun/2019:02:40:57 +0000] "POST /v2/compute/projects/2040f60a-7ffa-4263-b3df-ae126169ce27/docker/nodes HTTP/1.1" 409 436 "-" "Python/3.6 aiohttp/3.5.4"
2019-06-03 02:41:28 INFO web_log.py:233 192.168.0.211 [03/Jun/2019:02:40:57 +0000] "POST /v2/projects/2040f60a-7ffa-4263-b3df-ae126169ce27/templates/5d97d3cd-7b43-43f6-b488-b3d78f954bbd HTTP/1.1" 409 407 "-" "GNS3 QT Client v2.2.0b2"
2019-06-03 02:41:28 INFO web_log.py:233 192.168.0.211 [03/Jun/2019:02:41:28 +0000] "GET /v2/computes HTTP/1.1" 200 1024 "-" "GNS3 QT Client v2.2.0b2"`

[32e3e32]

In case it helps, in GNS 2.1.20 I was able to add the docker image and create a container via GNS GUI without any errors.

[adosztal]

I've just deployed a fresh install of 2.2.0b2 and observed the same. The container image is obviously the same, I checked the Docker version, that's the same as well:

# My Linux box with GNS3 2.1.20
adosztal@lab:~/Documents/gns3-testing$ docker --version
Docker version 18.09.6, build 481bc77

# Test environment
vagrant@gns3-testing-server:~$ docker --version
Docker version 18.09.6, build 481bc77

It must be something with the GNS3 server.

[grossmj]

I confirm I can reproduce the bug on my side.

[grossmj]

Looks like the problem comes from the additional persistent docker volumes support.

0c19bc8#diff-4ab6e8dce049256f913125a556166e6c

[grossmj]

The implementation was done in #1584 @kazkansouh Do you mind having a look at this? thanks

[kazkansouh]

Hello. Just had a quick look and it appears the issue is that the image marks /etc as a volume and GNS3 internally marks /etc/network as a volume (for configuring the network interface). The check was intended on a sanity check on user input to make sure that they do not have nested volumes. But sadly in this case it appears to be too restrictive.

The code could be changed to select the most general volume, i.e. in this case it would select /etc and silently ignore /etc/network?

[kazkansouh]

Or disable the check completely ...

[grossmj]

@kazkansouh thanks for looking at this, I will see what solution make more sense 👍

[kazkansouh]

I have taken a closer look at this issue and why adosztal/net_toolbox was working before without causing issues.

First, I am using:

$ docker --version
Docker version 18.09.2, build 6247962

Below are some tests in how Docker behaves with nested mounts.

1. It appears that Docker allows nested mounts. However, as a side effect can create empty directories on the host. It does not matter in which order the -v arguments are given.

    $ stat test
    stat: cannot stat 'test': No such file or directory
    
    $ docker run -t -i --rm -v $(pwd)/:/test/test -v $(pwd)/:/test alpine:3.9 /bin/sh -c "mount | grep test"
    /dev/nvme0n1p7 on /test type ext4 (rw,relatime,data=ordered)
    /dev/nvme0n1p7 on /test/test type ext4 (rw,relatime,data=ordered)
    
    $ stat test
      File: test
      Size: 4096            Blocks: 8          IO Block: 4096   directory
    Device: 10307h/66311d   Inode: 1933958     Links: 2
    Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-06-04 21:27:18.236998741 +0100
    Modify: 2019-06-04 21:27:18.236998741 +0100
    Change: 2019-06-04 21:27:18.236998741 +0100
     Birth: -
   

   Notice, the directory test on the host is now owned by root :(

2. When the same volume is mounted twice (with same host path), it appears as if it was only provided once:

    $ docker run -t -i --rm -v $(pwd)/:/test -v $(pwd)/:/test alpine:3.9 /bin/sh -c "mount | grep test"
    /dev/nvme0n1p7 on /test type ext4 (rw,relatime,data=ordered)
   

   However, this does not work when using the Docker API in GNS3 (with the checks disabled):

   

3. However, if the host path changes then it is an error:

    $ docker run -t -i --rm -v $(pwd)/tmp:/test -v $(pwd)/:/test alpine:3.9 true
    docker: Error response from daemon: Duplicate mount point: /test.
    See 'docker run --help'.
   

Moreover, it looks like adosztal/net_toolbox was mounting both /etc and /etc/network, and then (in GNS3 docker init script) binding both to their correct location. See following excerpt from docker inspect:

        "Mounts": [
            {
                "Type": "bind",
                "Source": "/opt/gns3/projects/z-docker-error/project-files/docker/fbf72ee9-db61-47dd-9b50-ed033e78653f/etc/network",
                "Destination": "/gns3volumes/etc/network",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/opt/gns3/projects/z-docker-error/project-files/docker/fbf72ee9-db61-47dd-9b50-ed033e78653f/etc",
                "Destination": "/gns3volumes/etc",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },

And from within the container:

root@adosztal-net_toolbox-1:~# mount | grep etc
/dev/nvme0n1p5 on /etc type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda1 on /gns3volumes/etc type ext4 (rw,nosuid,nodev,relatime,stripe=8191,data=ordered)
/dev/nvme0n1p5 on /etc/hostname type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda1 on /gns3volumes/etc/network type ext4 (rw,nosuid,nodev,relatime,stripe=8191,data=ordered)
/dev/sda1 on /etc/network type ext4 (rw,nosuid,nodev,relatime,stripe=8191,data=ordered)
/dev/sda1 on /etc type ext4 (rw,nosuid,nodev,relatime,stripe=8191,data=ordered)

Thus, the condition in the code that each volume must not encapsulate any other is not needed, i.e. it is only required that each mount point is unique. That said, having encapsulated mounts does not add any benefit as they are all mapped into project-files.

My preference would be to remove the error message that triggered this issue and automatically select the most general volume (or if they are the same select only one of them). @grossmj if its of use I can put together a PR for this?

[grossmj]

@grossmj if its of use I can put together a PR for this?

Thanks for the detailed explanation. Please feel free to put together a PR for this 👍

[adosztal]

I used /etc only because there were a couple subdirectories I needed and this way was easier but I can specify those to remove the error.
Which one do you guys prefer? Automatically selecting the most general volume prevents issues like this in the future; on the other hand changing my Dockerfile is a quick & easy fix.

[grossmj]

Fixed by #1597