Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues creating containers & links with basic topology #370

Closed
qlyoung opened this issue Apr 8, 2021 · 9 comments
Closed

Issues creating containers & links with basic topology #370

qlyoung opened this issue Apr 8, 2021 · 9 comments

Comments

@qlyoung
Copy link

qlyoung commented Apr 8, 2021

This project is extremely cool. Thank you for building it.

I took it for a spin and hit a few issues I'd appreciate some help with.


Environment: Debian 10 (buster)
Kernel: Linux buster 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
Arch: x64
Docker version:

root@buster /h/v/lab# docker version
Client:
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.11.6
 Git commit:        4c52b90
 Built:             Sun, 21 Feb 2021 18:18:35 +0100
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.6
  Git commit:       4c52b90
  Built:            Sun Feb 21 17:18:35 2021
  OS/Arch:          linux/amd64
  Experimental:     false

containerlab version:

version: 0.12.0
 commit: abadb94
   date: 2021-03-29T07:42:16Z
 source: https://github.com/srl-labs/containerlab

This is a stock Debian 10 VM running under Virtualbox.


Description

I wrote the following topology file:

name: pim-basic
topology:
        nodes:
                rp:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                fhr:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                lhr:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                source:
                        kind: linux
                        image: alpine:latest
                consumer:
                        kind: linux
                        image: alpine:latest
        links:
                - endpoints: ["fhr:eth0", "rp:eth0"]
                - endpoints: ["lhr:eth0", "rp:eth1"]

I then run containerlab deploy --topo lab.yaml.

Here's the output:

INFO[0000] Parsing & checking topology file: lab.yaml
INFO[0000] Creating lab directory: /home/vagrant/lab/clab-pim-basic
INFO[0000] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0/24', IPv6Subnet='2001:172:20:20::/80', MTU=
'1500'
INFO[0000] Creating container: source
INFO[0000] Creating container: consumer
INFO[0000] Creating container: rp
INFO[0000] Creating container: fhr
INFO[0000] Creating container: lhr
ERRO[0001] failed to create node source: Error response from daemon: OCI runtime create failed: container_linux.go:3
44: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407: running p
restart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2021-04-08T06:18:13Z\\\\
\\\" level=fatal msg=\\\\\\\"failed to add interface veth134fd21 to sandbox: error setting interface \\\\\\\\\\\\\\\
"veth134fd21\\\\\\\\\\\\\\\" IPv6 to 2001:172:20:20::3/80: failed to enable ipv6: reexec to set IPv6 failed: exit st
atus 2\\\\\\\"\\\\n\\\"\"": unknown
INFO[0001] Creating virtual wire: fhr:eth0 <--> rp:eth0
INFO[0001] Creating virtual wire: lhr:eth0 <--> rp:eth1
ERRO[0001] failed to rename link: file exists
ERRO[0001] failed to rename link: file exists
ERRO[0002] failed to run postdeploy task for node source: failed to Statfs "": no such file or directory
INFO[0002] Writing /etc/hosts file
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| # |          Name           | Container ID |        Image         | Kind  | Group |  State  |  IPv4 Address  |     IPv6 Address     |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| 1 | clab-pim-basic-consumer | 10cfdd78e2cd | alpine:latest        | linux |       | running | 172.20.20.4/24 | 2001:172:20:20::4/80 |
| 2 | clab-pim-basic-fhr      | 88120926affe | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.5/24 | 2001:172:20:20::5/80 |
| 3 | clab-pim-basic-lhr      | 01c8b8cdd1b3 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.6/24 | 2001:172:20:20::6/80 |
| 4 | clab-pim-basic-rp       | 9069fb4f1dc1 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.2/24 | 2001:172:20:20::2/80 |
| 5 | clab-pim-basic-source   | f935e7829062 | alpine:latest        | linux |       | created | 172.20.20.3/24 | 2001:172:20:20::3/80 |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+

The issues I noticed:

  • I notice some error logs; are these normal?
  • rp has two interfaces defined in the topology, but I only see one veth created in the running container:
root@buster /h/v/lab# docker exec -it clab-pim-basic-rp sh
/ # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
46: eth0@if47: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:14:14:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
  • source is reported as created in the initial deploy output, and in inspect output, and does not appear in docker ps and is in fact not running. I've seen this happen with fhr on another attempt so it seems to be random which container doesn't make it:
root@buster /h/v/lab# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
01c8b8cdd1b3        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   18 minutes ago      Up 18 minutes                           clab-pim-basic-lhr
88120926affe        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   18 minutes ago      Up 18 minutes                           clab-pim-basic-fhr
10cfdd78e2cd        alpine:latest          "/bin/sh"                18 minutes ago      Up 18 minutes                           clab-pim-basic-consumer
9069fb4f1dc1        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   18 minutes ago      Up 18 minutes                           clab-pim-basic-rp

Am I doing something wrong? Let me know if you need any additional info.

@qlyoung qlyoung changed the title Issues creating containers with basic topology Issues creating containers & links with basic topology Apr 8, 2021
@hellt
Copy link
Member

hellt commented Apr 8, 2021

Hi @qlyoung
thanks for checking clab out

Before going into the errors themselves, I wanted to point out some issues I see with the endpoints configuration:

links:
    - endpoints: ["fhr:eth0", "rp:eth0"]
    - endpoints: ["lhr:eth0", "rp:eth1"]

You shouldn't have eth0 interfaces in the endpoints sections. eth0 interface is automatically created by docker to connect a container to a management network.

image

If your intention was to interconnect fhr with rp over their data interfaces, then you would need to do the following

links:
  - endpoints: ["fhr:eth1", "rp:eth1"]

if, on the other hand, you wanted all the nodes to share the management network, then no links section is even needed.

Let me know what was the topology you wanted to build and we can sort it out :)

@qlyoung
Copy link
Author

qlyoung commented Apr 8, 2021

Ok, I'll switch to eth1 and see if the issues go away.

@qlyoung
Copy link
Author

qlyoung commented Apr 8, 2021

New topology:

root@buster /h/v/lab# cat lab.yaml
name: pim-basic
topology:
        nodes:
                rp:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                fhr:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                lhr:
                        kind: linux
                        image: frrouting/frr:v7.5.0
                source:
                        kind: linux
                        image: alpine:latest
                consumer:
                        kind: linux
                        image: alpine:latest
        links:
                - endpoints: ["fhr:eth1", "rp:eth1"]
                - endpoints: ["lhr:eth1", "rp:eth2"]

This summarizes the topology I want for now, just lhr -- rp -- fhr. However I still expect the other containers to spin up.

New output:

root@buster /h/v/lab# containerlab deploy --topo ./lab.yaml
INFO[0000] Parsing & checking topology file: lab.yaml
INFO[0000] Creating lab directory: /home/vagrant/lab/clab-pim-basic
INFO[0000] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0/24', IPv6Subnet='2001:172:20:20::/80', MTU='1500'
INFO[0000] Creating container: source
INFO[0000] Creating container: lhr
INFO[0000] Creating container: consumer
INFO[0000] Creating container: rp
INFO[0000] Creating container: fhr
ERRO[0001] failed to create node source: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: time=\\\\\\\"2021-04-08T15:49:56Z\\\\\\\" level=fatal msg=\\\\\\\"failed to add interface vethf4b055f to sandbox: error setting interface \\\\\\\\\\\\\\\"vethf4b055f\\\\\\\\\\\\\\\" IPv6 to 2001:172:20:20::2/80: failed to enable ipv6: reexec to set IPv6 failed: exit status 2\\\\\\\"\\\\n\\\"\"": unknown
INFO[0001] Creating virtual wire: fhr:eth1 <--> rp:eth1
INFO[0001] Creating virtual wire: lhr:eth1 <--> rp:eth2
ERRO[0001] failed to run postdeploy task for node source: failed to Statfs "": no such file or directory
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| # |          Name           | Container ID |        Image         | Kind  | Group |  State  |  IPv4 Address  |     IPv6 Address     |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| 1 | clab-pim-basic-consumer | 8dd29f657b6e | alpine:latest        | linux |       | running | 172.20.20.3/24 | 2001:172:20:20::3/80 |
| 2 | clab-pim-basic-fhr      | 671f5ca9a60c | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.4/24 | 2001:172:20:20::4/80 |
| 3 | clab-pim-basic-lhr      | 2f066810e60d | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.5/24 | 2001:172:20:20::5/80 |
| 4 | clab-pim-basic-rp       | e10a90e91d27 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.6/24 | 2001:172:20:20::6/80 |
| 5 | clab-pim-basic-source   | 7131aab38f3c | alpine:latest        | linux |       | created | 172.20.20.2/24 | 2001:172:20:20::2/80 |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+


root@buster /h/v/lab# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
e10a90e91d27        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   8 seconds ago       Up 6 seconds                            clab-pim-basic-rp
2f066810e60d        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   8 seconds ago       Up 6 seconds                            clab-pim-basic-lhr
8dd29f657b6e        alpine:latest          "/bin/sh"                8 seconds ago       Up 6 seconds                            clab-pim-basic-consumer
671f5ca9a60c        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   8 seconds ago       Up 6 seconds                            clab-pim-basic-fhr

Looks like rp has its interfaces up now:

root@buster /h/v/lab# docker exec -it clab-pim-basic-rp sh
/ # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
69: eth0@if70: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:14:14:06 brd ff:ff:ff:ff:ff:ff link-netnsid 0
71: eth1@if72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue state UP mode DEFAULT group default
    link/ether 7a:19:ff:12:06:23 brd ff:ff:ff:ff:ff:ff link-netnsid 1
73: eth2@if74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue state UP mode DEFAULT group default
    link/ether a2:01:9d:48:79:88 brd ff:ff:ff:ff:ff:ff link-netnsid 2

So now the only issues are

  1. The error output in the deploy
  2. Failure to start one or more of the containres

For 2, I destroyed the lab, restarted Docker, and spun up the lab again. This time two containers are missing, fhr and source:

+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| # |          Name           | Container ID |        Image         | Kind  | Group |  State  |  IPv4 Address  |     IPv6 Address     |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
| 1 | clab-pim-basic-consumer | 727c2eadb896 | alpine:latest        | linux |       | running | 172.20.20.5/24 | 2001:172:20:20::5/80 |
| 2 | clab-pim-basic-fhr      | aa22f7622aa9 | frrouting/frr:v7.5.0 | linux |       | created | 172.20.20.4/24 | 2001:172:20:20::4/80 |
| 3 | clab-pim-basic-lhr      | b2d92d753cd2 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.3/24 | 2001:172:20:20::3/80 |
| 4 | clab-pim-basic-rp       | c4c3b0eb5770 | frrouting/frr:v7.5.0 | linux |       | running | 172.20.20.2/24 | 2001:172:20:20::2/80 |
| 5 | clab-pim-basic-source   | 17a54d4c97b4 | alpine:latest        | linux |       | created | 172.20.20.6/24 | 2001:172:20:20::6/80 |
+---+-------------------------+--------------+----------------------+-------+-------+---------+----------------+----------------------+
root@buster /h/v/lab# docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS               NAMES
b2d92d753cd2        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   6 seconds ago       Up 3 seconds                            clab-pim-basic-lhr
727c2eadb896        alpine:latest          "/bin/sh"                6 seconds ago       Up 3 seconds                            clab-pim-basic-consumer
c4c3b0eb5770        frrouting/frr:v7.5.0   "/sbin/tini -- /usr/…"   6 seconds ago       Up 3 seconds                            clab-pim-basic-rp

@qlyoung
Copy link
Author

qlyoung commented Apr 8, 2021

If it helps you can spin up the same VM with

vagrant init debian/buster64
vagrant up

https://app.vagrantup.com/debian/boxes/buster64

I installed containerlab with

bash -c "$(curl -sL https://get-clab.srlinux.dev)"

and docker with

apt install docker.io

@hellt
Copy link
Member

hellt commented Apr 8, 2021 via email

@qlyoung
Copy link
Author

qlyoung commented Apr 8, 2021

The docker-ce package is the one provided by docker themselves; docker.io is the one provided by distributions. I would expect clab to work with distro docker packages. I'll try using docker-ce and report back.

@hellt
Copy link
Member

hellt commented Apr 8, 2021 via email

@qlyoung
Copy link
Author

qlyoung commented Apr 8, 2021

I tried with docker-ce on a new VM, and it failed in a different way. One that showed me what the real problem is:

root@buster:/home/vagrant/lab# free -m
              total        used        free      shared  buff/cache   available
Mem:            483         145         185           2         152         322
Swap:             0           0           0

By default this VM image has 512 mb of memory.

I went back to the initial environment, increased memory to 4gb, and the errors disappeared.

However, I feel the error output should have included something about OOM. If that's not feasible it might be prudent to make a note of what OOM failures look like, since it's not really clear (not seeing the usual Killed). I did read the footnote on the installation page indicating that memory usage is dependent on lab size.

Shall I close this or leave it open for tracking that?

@hellt
Copy link
Member

hellt commented Apr 8, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants