Issue deploying SRLinux clab on 0.51

[sacckth]

Following lab fails to be deployed. without "links" nodes come up ok

❯ cat clab.yaml

name: srl_lab
topology:
  kinds:
    nokia_srlinux:
      image: harbor.ncse.io/library/srlinux:23.10.2
      license: ./license.key
  nodes:
    srl1:
      kind: nokia_srlinux
    srl2:
      kind: nokia_srlinux
      type: ixr6e
  links:
    - endpoints: ["srl1:e1-1-1","srl2:e1-1-1"]

❯ sudo clab dep -t clab.yaml -ddd
INFO[0000] Containerlab v0.51.0 started
DEBU[0000] template variables: <nil>
DEBU[0000] topology:
name: srl_lab
topology:
  kinds:
    nokia_srlinux:
      image: harbor.ncse.io/library/srlinux:23.10.2
      license: ./license.key
  nodes:
    srl1:
      kind: nokia_srlinux
    srl2:
      kind: nokia_srlinux
      type: ixr6e
  links:
    - endpoints: ["srl1:e1-1-1","srl2:e1-1-1"]

DEBU[0000] method initMgmtNetwork was called mgmt params &{Network: Bridge: IPv4Subnet: IPv4Gw: IPv4Range: IPv6Subnet: IPv6Gw: IPv6Range: MTU:0 ExternalAccess:<nil>}
DEBU[0000] New mgmt params are &{Network:clab Bridge: IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv4Range: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: IPv6Range: MTU:0 ExternalAccess:0xc0005446ea}
DEBU[0000] env runtime var value is
DEBU[0000] Running runtime.Init with params &{Timeout:2m0s GracefulShutdown:false Debug:false KeepMgmtNet:false VerifyLinkParams:<nil>} and &{Network:clab Bridge: IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv4Range: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: IPv6Range: MTU:0 ExternalAccess:0xc0005446ea}
DEBU[0000] Runtime: Docker
DEBU[0000] detected docker network mtu value - 1500
DEBU[0000] detected network name in use: clab, backed by a bridge br-7dec063fb80a
DEBU[0000] initialized a runtime with params &{config:{Timeout:120000000000 GracefulShutdown:false Debug:false KeepMgmtNet:false VerifyLinkParams:0xc0010e816f} Client:0xc0010ee100 mgmt:0xc00052e000}
INFO[0000] Parsing & checking topology file: clab.yaml
DEBU[0000] node config: &{ShortName:srl1 LongName:clab-srl_lab-srl1 Fqdn:srl1.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 Index:0 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:<nil> ResConfig: NodeType: Position: License: Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc00052b170 Healthcheck:<nil> NSPath: Publish:[] ExtraHosts:[] Labels:map[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc00052b1a0 DNS:0xc000a68050 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0000] node config: &{ShortName:srl2 LongName:clab-srl_lab-srl2 Fqdn:srl2.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 Index:1 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:<nil> ResConfig: NodeType:ixr6e Position: License: Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc00052b9e0 Healthcheck:<nil> NSPath: Publish:[] ExtraHosts:[] Labels:map[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc00052ba10 DNS:0xc000a68500 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0000] lab Conf: &{Name:srl_lab Prefix:0xc000c60660 Mgmt:0xc00052e000 Settings:<nil> Topology:0xc00052a0c0 Debug:false}
DEBU[0000] Env: CLAB_VERSION_CHECK=
DEBU[0000] Checking if docker network "clab" exists
DEBU[0000] network "clab" was found. Reusing it...
DEBU[0000] Docker network "clab", bridge name "br-7dec063fb80a"
DEBU[0000] Disable RPF check on the docker host
DEBU[0000] Enable LLDP on the linux bridge br-7dec063fb80a
DEBU[0000] Disabling TX checksum offloading for the br-7dec063fb80a bridge interface...
DEBU[0000] using nf_tables as the firewall interface
DEBU[0000] found iptables forwarding rule targeting the bridge "br-7dec063fb80a". Skipping creation of the forwarding rule.
DEBU[0000] kernel version: 5.15.0-94-generic
DEBU[0000] Looking up harbor.ncse.io/library/srlinux:23.10.2 Docker image
DEBU[0000] Image harbor.ncse.io/library/srlinux:23.10.2 present, skip pulling
DEBU[0000] kernel version: 5.15.0-94-generic
DEBU[0000] Looking up harbor.ncse.io/library/srlinux:23.10.2 Docker image
DEBU[0000] Image harbor.ncse.io/library/srlinux:23.10.2 present, skip pulling
Error: containers ["clab-srl_lab-srl1" "clab-srl_lab-srl2"] already exist. Add '--reconfigure' flag to the deploy command to first remove the containers and then deploy the lab
❯ sudo clab dep -t clab.yaml -ddd --reconfigure
INFO[0000] Containerlab v0.51.0 started
DEBU[0000] template variables: <nil>
DEBU[0000] topology:
name: srl_lab
topology:
  kinds:
    nokia_srlinux:
      image: harbor.ncse.io/library/srlinux:23.10.2
      license: ./license.key
  nodes:
    srl1:
      kind: nokia_srlinux
    srl2:
      kind: nokia_srlinux
      type: ixr6e
  links:
    - endpoints: ["srl1:e1-1-1","srl2:e1-1-1"]

DEBU[0000] method initMgmtNetwork was called mgmt params &{Network: Bridge: IPv4Subnet: IPv4Gw: IPv4Range: IPv6Subnet: IPv6Gw: IPv6Range: MTU:0 ExternalAccess:<nil>}
DEBU[0000] New mgmt params are &{Network:clab Bridge: IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv4Range: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: IPv6Range: MTU:0 ExternalAccess:0xc00102a76a}
DEBU[0000] env runtime var value is
DEBU[0000] Running runtime.Init with params &{Timeout:2m0s GracefulShutdown:false Debug:false KeepMgmtNet:false VerifyLinkParams:<nil>} and &{Network:clab Bridge: IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv4Range: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: IPv6Range: MTU:0 ExternalAccess:0xc00102a76a}
DEBU[0000] Runtime: Docker
DEBU[0000] detected docker network mtu value - 1500
DEBU[0000] detected network name in use: clab, backed by a bridge br-7dec063fb80a
DEBU[0000] initialized a runtime with params &{config:{Timeout:120000000000 GracefulShutdown:false Debug:false KeepMgmtNet:false VerifyLinkParams:0xc00022c14f} Client:0xc000f83e00 mgmt:0xc0005298c0}
INFO[0000] Parsing & checking topology file: clab.yaml
DEBU[0000] node config: &{ShortName:srl1 LongName:clab-srl_lab-srl1 Fqdn:srl1.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 Index:0 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:<nil> ResConfig: NodeType: Position: License: Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc0005763f0 Healthcheck:<nil> NSPath: Publish:[] ExtraHosts:[] Labels:map[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc000576420 DNS:0xc000534050 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0000] node config: &{ShortName:srl2 LongName:clab-srl_lab-srl2 Fqdn:srl2.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 Index:1 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:<nil> ResConfig: NodeType:ixr6e Position: License: Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc000576c60 Healthcheck:<nil> NSPath: Publish:[] ExtraHosts:[] Labels:map[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc000576c90 DNS:0xc000534730 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0000] lab Conf: &{Name:srl_lab Prefix:0xc000cc0390 Mgmt:0xc0005298c0 Settings:<nil> Topology:0xc000b54750 Debug:false}
DEBU[0000] Env: CLAB_VERSION_CHECK=
DEBU[0000] Filter key: name, filter value: ^clab-srl_lab-srl1$
DEBU[0000] Filter key: name, filter value: ^clab-srl_lab-srl2$
INFO[0000] Destroying lab: srl_lab
DEBU[0000] failed to Statfs "/proc/0/ns/net": no such file or directory
DEBU[0000] Removing container: clab-srl_lab-srl1
DEBU[0000] Removing container: clab-srl_lab-srl2
INFO[0000] Removed container: clab-srl_lab-srl2
DEBU[0000] Worker 0 terminating...
DEBU[0000] latest version 0.51.3 is newer than the current one 0.51.0
INFO[0000] Removed container: clab-srl_lab-srl1
DEBU[0000] Worker 1 terminating...
INFO[0000] Removing containerlab host entries from /etc/hosts file
INFO[0000] Removing ssh config for containerlab nodes
DEBU[0000] Calling DeleteNet method. *CLab.Config.Mgmt value is: &{Network:clab Bridge:br-7dec063fb80a IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv4Range: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: IPv6Range: MTU:1500 ExternalAccess:0xc00102a76a}
DEBU[0000] removing clab iptables rules for bridge "br-7dec063fb80a"
DEBU[0000] Deleting clab-srl_lab-srl2 network namespace
DEBU[0000] Deleting netns symlink: clab-srl_lab-srl2
DEBU[0000] Deleting clab-srl_lab-srl1 network namespace
DEBU[0000] Deleting netns symlink: clab-srl_lab-srl1
INFO[0000] Removing /usr/local/bugatti9/tmp/clab-srl_lab directory...
DEBU[0000] Checking if docker network "clab" exists
DEBU[0000] Network "clab" does not exist
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU='ל'
DEBU[0000] bridge "br-7dec063fb80a" has ipv4 addr of "" and ipv6 addr of ""
DEBU[0000] Docker network "clab", bridge name "br-7dec063fb80a"
DEBU[0000] Disable RPF check on the docker host
DEBU[0000] Enable LLDP on the linux bridge br-7dec063fb80a
DEBU[0000] Disabling TX checksum offloading for the br-7dec063fb80a bridge interface...
DEBU[0000] using nf_tables as the firewall interface
DEBU[0000] Installing iptables rules for bridge "br-7dec063fb80a"
DEBU[0000] kernel version: 5.15.0-94-generic
DEBU[0000] Looking up harbor.ncse.io/library/srlinux:23.10.2 Docker image
DEBU[0000] Image harbor.ncse.io/library/srlinux:23.10.2 present, skip pulling
DEBU[0000] kernel version: 5.15.0-94-generic
DEBU[0000] Looking up harbor.ncse.io/library/srlinux:23.10.2 Docker image
DEBU[0000] Image harbor.ncse.io/library/srlinux:23.10.2 present, skip pulling
DEBU[0000] kernel module "ip_tables" is already loaded
DEBU[0000] kernel module "ip6_tables" is already loaded
INFO[0000] Creating lab directory: /usr/local/bugatti9/tmp/clab-srl_lab
DEBU[0000] Existing entry "Tag:       MASK (16), ID: 4294967295, Perm: rwx (7)" deleted
DEBU[0000] writing cert file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/ca/ca.pem
DEBU[0000] writing key file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/ca/ca.key
DEBU[0000] user home dir /home/schavezc found using os/user.LookupId
DEBU[0000] user home dir /home/schavezc found using os/user.LookupId
DEBU[0000] /home/schavezc/.ssh/authorized_keys found, adding it to the list of files to get public keys from
DEBU[0000] Loaded public key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8WlbqK5AF5tOd9S5lSfXxD1NIK4KhbltWJs9D9JGIFQjsUaZwR813JIIv+uwiOYR7YmZr29RezTTlWZ82xmj0V3Ap+PB5+2SiM0aV2kyeGpwIO4SZQMq7sWO3gz0MlM5HJPYc9QppG012n/O/bgl29TSBmlMqnIXzf3oPNxf3aJkYqE7p+9Z30+LG8FGd+l8hYR7jvPJ9BZxluOI5xV/wI+y0Dsx5V8OBblhQluo4bthVOr5E/N0hWkJD3wYZWgUgcOw4JFC/2FsDhwJPUqMDFRNGOC3GW7Fk6YufBKS95AoKVCFP06NCjsy7mVX/K7L0pOsPNmvRDMBBPTppZpX0p2g0OkNY5T0F0EZhzeYv5+oi9o9l5wBRlKpAVHJmyz8aZIIoquuiVMXDFLMaEnEaEB31OuQn47aDt/q0DzB43QE6RyxXHnUvQ6tqlZehVRvZB24/+cBzHq8Q7pb4m2saimUZF09WK1cBbc8zKoJZPhAkgYjySPf9fKfBfAmPGMbqUbVgolK1afZL5QDN8Zf1BVa4QktcRZ6zRdn8jdqWSo+G7gRnVKnGq1nNedm/0eOi2xFgwXljr2+uoKr0igp8C9wRv65+DuTOetx5qwEySaxcelaal3c4EIsIar3J+E0eZJspgeJugnGvhLnU62kDUiqo/oa1gdGGVrAMhb8+Nw== schavez@bugatti9
DEBU[0000] Loaded public key ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEhohE89FYbVjan9Uh19vXPQzdyDUgJSlY9D6Y4BW6W1 schavezc
DEBU[0000] SSH_AUTH_SOCK not set, skipping pubkey fetching
DEBU[0000] Number of Node workers: 2, Number of Link workers: 2
DEBU[0000] Dependencies:
srl1 -> [  ]
srl2 -> [  ]
DEBU[0000] - cycle check round 1 -
srl1 <- [  ]
srl2 <- [  ]
DEBU[0000] node creation graph is successfully validated as being acyclic
DEBU[0000] Stage Change: Enter Wait -> srl2 - create
DEBU[0000] Stage Change: Enter Go -> srl2 - create
DEBU[0000] Stage Change: Enter Wait -> srl1 - create
DEBU[0000] Stage Change: Enter Go -> srl1 - create
DEBU[0000] Worker 0 received node: &{ShortName:srl2 LongName:clab-srl_lab-srl2 Fqdn:srl2.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 Index:1 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:0xc000266060 ResConfig: NodeType:ixr6e Position: License:/usr/local/bugatti9/tmp/license.key Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[net.ipv4.ip_forward:0 net.ipv6.conf.all.accept_dad:0 net.ipv6.conf.all.autoconf:0 net.ipv6.conf.all.disable_ipv6:0 net.ipv6.conf.default.accept_dad:0 net.ipv6.conf.default.autoconf:0] User:0:0 Entrypoint: Cmd:sudo bash -c 'touch /.dockerenv && /opt/srlinux/bin/sr_linux' Exec:[] Env:map[CLAB_INTFS:1 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:nokia_srlinux CLAB_LABEL_CLAB_NODE_LAB_DIR:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 CLAB_LABEL_CLAB_NODE_NAME:srl2 CLAB_LABEL_CLAB_NODE_TYPE:ixr6e CLAB_LABEL_CLAB_TOPO_FILE:/usr/local/bugatti9/tmp/clab.yaml CLAB_LABEL_CONTAINERLAB:srl_lab SRLINUX:1] Binds:[/usr/local/bugatti9/tmp/clab-srl_lab/srl2/license.key:/opt/srlinux/etc/license.key:ro /usr/local/bugatti9/tmp/clab-srl_lab/srl2/config:/etc/opt/srlinux/:rw /usr/local/bugatti9/tmp/clab-srl_lab/srl2/topology.yml:/tmp/topology.yml:ro] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc000576c60 Healthcheck:<nil> NSPath:/proc/0/ns/net Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:br-7dec063fb80a clab-node-group: clab-node-kind:nokia_srlinux clab-node-lab-dir:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 clab-node-name:srl2 clab-node-type:ixr6e clab-topo-file:/usr/local/bugatti9/tmp/clab.yaml containerlab:srl_lab] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc000576c90 DNS:0xc000534730 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0000] Creating directory structure for SRL container: srl2
DEBU[0000] Worker 1 received node: &{ShortName:srl1 LongName:clab-srl_lab-srl1 Fqdn:srl1.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 Index:0 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:0xc00052e010 ResConfig: NodeType:ixrd2 Position: License:/usr/local/bugatti9/tmp/license.key Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[net.ipv4.ip_forward:0 net.ipv6.conf.all.accept_dad:0 net.ipv6.conf.all.autoconf:0 net.ipv6.conf.all.disable_ipv6:0 net.ipv6.conf.default.accept_dad:0 net.ipv6.conf.default.autoconf:0] User:0:0 Entrypoint: Cmd:sudo bash -c 'touch /.dockerenv && /opt/srlinux/bin/sr_linux' Exec:[] Env:map[CLAB_INTFS:1 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:nokia_srlinux CLAB_LABEL_CLAB_NODE_LAB_DIR:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 CLAB_LABEL_CLAB_NODE_NAME:srl1 CLAB_LABEL_CLAB_NODE_TYPE:ixrd2 CLAB_LABEL_CLAB_TOPO_FILE:/usr/local/bugatti9/tmp/clab.yaml CLAB_LABEL_CONTAINERLAB:srl_lab SRLINUX:1] Binds:[/usr/local/bugatti9/tmp/clab-srl_lab/srl1/license.key:/opt/srlinux/etc/license.key:ro /usr/local/bugatti9/tmp/clab-srl_lab/srl1/config:/etc/opt/srlinux/:rw /usr/local/bugatti9/tmp/clab-srl_lab/srl1/topology.yml:/tmp/topology.yml:ro] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc0005763f0 Healthcheck:<nil> NSPath:/proc/315983/ns/net Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:br-7dec063fb80a clab-node-group: clab-node-kind:nokia_srlinux clab-node-lab-dir:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 clab-node-name:srl1 clab-node-type:ixrd2 clab-topo-file:/usr/local/bugatti9/tmp/clab.yaml containerlab:srl_lab] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc000576420 DNS:0xc000534050 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0000] Creating directory structure for SRL container: srl1
DEBU[0000] CopyFile src /usr/local/bugatti9/tmp/license.key -> dst /usr/local/bugatti9/tmp/clab-srl_lab/srl1/license.key succeeded
DEBU[0000] CopyFile src /usr/local/bugatti9/tmp/license.key -> dst /usr/local/bugatti9/tmp/clab-srl_lab/srl2/license.key succeeded
DEBU[0000] {1a:22:00:00:00:00}/usr/local/bugatti9/tmp/clab-srl_lab/srl1/topology.yml
DEBU[0000] {1a:0d:01:00:00:00}/usr/local/bugatti9/tmp/clab-srl_lab/srl2/topology.yml
DEBU[0000] configuration template for node srl1 is empty, skipping startup config file generation
INFO[0000] Creating container: "srl1"
DEBU[0000] configuration template for node srl2 is empty, skipping startup config file generation
INFO[0000] Creating container: "srl2"
DEBU[0001] Container "srl2" create response: {ID:0192fd081a900e0138f3487470836a61119851e2905db62c6c1ca2e834d79560 Warnings:[]}
DEBU[0001] Start container: "clab-srl_lab-srl2"
DEBU[0001] Container "srl1" create response: {ID:c6ed4c6dcce63af5f5c575aa72ea4a84456d5ef4a36083fa8b00c71469d61d52 Warnings:[]}
DEBU[0001] Start container: "clab-srl_lab-srl1"
DEBU[0001] Container started: "clab-srl_lab-srl2"
DEBU[0001] Filter key: name, filter value: ^clab-srl_lab-srl2$
DEBU[0001] StateChange: Done -> srl2 - create
DEBU[0001] Stage Change: Enter Wait -> srl2 - create-links
DEBU[0001] Stage Change: Enter Go -> srl2 - create-links
ERRO[0001] stage "srl2" not found
DEBU[0001] StateChange: Done -> srl2 - create-links
DEBU[0001] Stage Change: Enter Wait -> srl2 - configure
DEBU[0001] Stage Change: Enter Go -> srl2 - configure
INFO[0001] Running postdeploy actions for Nokia SR Linux 'srl2' node
DEBU[0001] creating node certificate for srl2
DEBU[0001] Container started: "clab-srl_lab-srl1"
DEBU[0001] Filter key: name, filter value: ^clab-srl_lab-srl1$
DEBU[0001] StateChange: Done -> srl1 - create
DEBU[0001] Stage Change: Enter Wait -> srl1 - create-links
DEBU[0001] Stage Change: Enter Go -> srl1 - create-links
INFO[0001] Creating link: srl1:e1-1-1 <--> srl2:e1-1-1
DEBU[0001] writing cert file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/srl2/srl2.pem
DEBU[0001] writing key file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/srl2/srl2.key
DEBU[0001] Method GetHostsPath was called with a resulting path "/var/lib/docker/containers/0192fd081a900e0138f3487470836a61119851e2905db62c6c1ca2e834d79560/hosts"
DEBU[0001] Waiting for SR Linux node "srl2" to boot...
DEBU[0001] /clab-srl_lab-srl2 exec created clab-srl_lab-srl2
DEBU[0001] /clab-srl_lab-srl2 exec attached clab-srl_lab-srl2
ERRO[0001] stage "srl1" not found
DEBU[0001] StateChange: Done -> srl1 - create-links
DEBU[0001] Stage Change: Enter Wait -> srl1 - configure
DEBU[0001] Stage Change: Enter Go -> srl1 - configure
INFO[0001] Running postdeploy actions for Nokia SR Linux 'srl1' node
DEBU[0001] creating node certificate for srl1
DEBU[0001] writing cert file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/srl1/srl1.pem
DEBU[0001] writing key file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/srl1/srl1.key
DEBU[0001] Method GetHostsPath was called with a resulting path "/var/lib/docker/containers/c6ed4c6dcce63af5f5c575aa72ea4a84456d5ef4a36083fa8b00c71469d61d52/hosts"
DEBU[0001] Waiting for SR Linux node "srl1" to boot...
DEBU[0001] /clab-srl_lab-srl1 exec created clab-srl_lab-srl1
DEBU[0001] /clab-srl_lab-srl1 exec attached clab-srl_lab-srl1
DEBU[0002] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:44:15.136 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0002] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:44:15.357 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0003] /clab-srl_lab-srl2 exec created clab-srl_lab-srl2
DEBU[0003] /clab-srl_lab-srl2 exec attached clab-srl_lab-srl2
DEBU[0003] /clab-srl_lab-srl1 exec created clab-srl_lab-srl1
DEBU[0003] /clab-srl_lab-srl1 exec attached clab-srl_lab-srl1
DEBU[0003] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:44:16.719 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0004] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:44:16.932 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0004] /clab-srl_lab-srl2 exec created clab-srl_lab-srl2
DEBU[0004] /clab-srl_lab-srl2 exec attached clab-srl_lab-srl2
DEBU[0005] /clab-srl_lab-srl1 exec created clab-srl_lab-srl1
DEBU[0005] /clab-srl_lab-srl1 exec attached clab-srl_lab-srl1
DEBU[0005] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:44:18.213 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0005] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "Parsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
ERRO[0006] failed to create exec in container /clab-srl_lab-srl2: Error response from daemon: container 0192fd081a900e0138f3487470836a61119851e2905db62c6c1ca2e834d79560 is not running
ERRO[0006] clab-srl_lab-srl2: failed to execute cmd: "/opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running" with error Error response from daemon: container 0192fd081a900e0138f3487470836a61119851e2905db62c6c1ca2e834d79560 is not running
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x25eee1b]

goroutine 169 [running]:
github.com/srl-labs/containerlab/clab/exec.(*ExecResult).GetReturnCode(...)
	github.com/srl-labs/containerlab/clab/exec/exec.go:141
github.com/srl-labs/containerlab/nodes/srl.(*srl).Ready(0xc001019900, {0x36cf0c8?, 0xc000534ff0?})
	github.com/srl-labs/containerlab/nodes/srl/srl.go:333 +0x1db
github.com/srl-labs/containerlab/nodes/srl.(*srl).PostDeploy(0xc001019900, {0x36cf0c8, 0xc000534ff0}, 0xc0005d6698)
	github.com/srl-labs/containerlab/nodes/srl/srl.go:262 +0x25a
github.com/srl-labs/containerlab/clab.(*CLab).scheduleNodes.func1(0xc000590778?, 0xc0015b0480, 0xc0005907d0?, {0x7b9386?, 0xc0005a3b00?})
	github.com/srl-labs/containerlab/clab/clab.go:589 +0x899
created by github.com/srl-labs/containerlab/clab.(*CLab).scheduleNodes
	github.com/srl-labs/containerlab/clab/clab.go:660 +0x177
❯ ls
clab-srl_lab  clab.yaml  license.key
❯ cat clab.yaml
name: srl_lab
topology:
  kinds:
    nokia_srlinux:
      image: harbor.ncse.io/library/srlinux:23.10.2
      license: ./license.key
  nodes:
    srl1:
      kind: nokia_srlinux
    srl2:
      kind: nokia_srlinux
      type: ixr6e
  links:
    - endpoints: ["srl1:e1-1-1","srl2:e1-1-1"]
❯ sudo clab dep -t clab.yaml -ddd --reconfigure
INFO[0000] Containerlab v0.51.0 started
DEBU[0000] template variables: <nil>
DEBU[0000] topology:
name: srl_lab
topology:
  kinds:
    nokia_srlinux:
      image: harbor.ncse.io/library/srlinux:23.10.2
      license: ./license.key
  nodes:
    srl1:
      kind: nokia_srlinux
    srl2:
      kind: nokia_srlinux
      type: ixr6e
  links:
    - endpoints: ["srl1:e1-1-1","srl2:e1-1-1"]

DEBU[0000] method initMgmtNetwork was called mgmt params &{Network: Bridge: IPv4Subnet: IPv4Gw: IPv4Range: IPv6Subnet: IPv6Gw: IPv6Range: MTU:0 ExternalAccess:<nil>}
DEBU[0000] New mgmt params are &{Network:clab Bridge: IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv4Range: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: IPv6Range: MTU:0 ExternalAccess:0xc00119c70a}
DEBU[0000] env runtime var value is
DEBU[0000] Running runtime.Init with params &{Timeout:2m0s GracefulShutdown:false Debug:false KeepMgmtNet:false VerifyLinkParams:<nil>} and &{Network:clab Bridge: IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv4Range: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: IPv6Range: MTU:0 ExternalAccess:0xc00119c70a}
DEBU[0000] Runtime: Docker
DEBU[0000] detected docker network mtu value - 1500
DEBU[0000] detected network name in use: clab, backed by a bridge br-7dec063fb80a
DEBU[0000] initialized a runtime with params &{config:{Timeout:120000000000 GracefulShutdown:false Debug:false KeepMgmtNet:false VerifyLinkParams:0xc00024e11f} Client:0xc0011f0100 mgmt:0xc001186000}
INFO[0000] Parsing & checking topology file: clab.yaml
DEBU[0000] node config: &{ShortName:srl1 LongName:clab-srl_lab-srl1 Fqdn:srl1.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 Index:0 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:<nil> ResConfig: NodeType: Position: License: Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc0011174d0 Healthcheck:<nil> NSPath: Publish:[] ExtraHosts:[] Labels:map[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc001117500 DNS:0xc0009ee050 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0000] node config: &{ShortName:srl2 LongName:clab-srl_lab-srl2 Fqdn:srl2.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 Index:1 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:<nil> ResConfig: NodeType:ixr6e Position: License: Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[] Binds:[] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc001182ed0 Healthcheck:<nil> NSPath: Publish:[] ExtraHosts:[] Labels:map[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc001182f00 DNS:0xc0009ee640 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0000] lab Conf: &{Name:srl_lab Prefix:0xc000bd81b0 Mgmt:0xc001186000 Settings:<nil> Topology:0xc0011820c0 Debug:false}
DEBU[0000] Env: CLAB_VERSION_CHECK=
DEBU[0000] Filter key: name, filter value: ^clab-srl_lab-srl1$
DEBU[0000] Filter key: name, filter value: ^clab-srl_lab-srl2$
INFO[0000] Destroying lab: srl_lab
DEBU[0000] failed to Statfs "/proc/0/ns/net": no such file or directory
DEBU[0000] Removing container: clab-srl_lab-srl2
DEBU[0000] Removing container: clab-srl_lab-srl1
INFO[0000] Removed container: clab-srl_lab-srl2
DEBU[0000] Worker 1 terminating...
DEBU[0000] latest version 0.51.3 is newer than the current one 0.51.0
INFO[0000] Removed container: clab-srl_lab-srl1
DEBU[0000] Worker 0 terminating...
INFO[0000] Removing containerlab host entries from /etc/hosts file
INFO[0000] Removing ssh config for containerlab nodes
DEBU[0000] Calling DeleteNet method. *CLab.Config.Mgmt value is: &{Network:clab Bridge:br-7dec063fb80a IPv4Subnet:172.20.20.0/24 IPv4Gw: IPv4Range: IPv6Subnet:2001:172:20:20::/64 IPv6Gw: IPv6Range: MTU:1500 ExternalAccess:0xc00119c70a}
DEBU[0000] removing clab iptables rules for bridge "br-7dec063fb80a"
DEBU[0000] Deleting clab-srl_lab-srl1 network namespace
DEBU[0000] Deleting netns symlink: clab-srl_lab-srl1
DEBU[0000] Deleting clab-srl_lab-srl2 network namespace
DEBU[0000] Deleting netns symlink: clab-srl_lab-srl2
INFO[0000] Removing /usr/local/bugatti9/tmp/clab-srl_lab directory...
DEBU[0000] Checking if docker network "clab" exists
DEBU[0000] Network "clab" does not exist
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU='ל'
DEBU[0000] bridge "br-7dec063fb80a" has ipv4 addr of "" and ipv6 addr of ""
DEBU[0000] Docker network "clab", bridge name "br-7dec063fb80a"
DEBU[0000] Disable RPF check on the docker host
DEBU[0000] Enable LLDP on the linux bridge br-7dec063fb80a
DEBU[0000] Disabling TX checksum offloading for the br-7dec063fb80a bridge interface...
DEBU[0000] using nf_tables as the firewall interface
DEBU[0000] Installing iptables rules for bridge "br-7dec063fb80a"
DEBU[0000] kernel version: 5.15.0-94-generic
DEBU[0000] Looking up harbor.ncse.io/library/srlinux:23.10.2 Docker image
DEBU[0000] Image harbor.ncse.io/library/srlinux:23.10.2 present, skip pulling
DEBU[0000] kernel version: 5.15.0-94-generic
DEBU[0000] Looking up harbor.ncse.io/library/srlinux:23.10.2 Docker image
DEBU[0000] Image harbor.ncse.io/library/srlinux:23.10.2 present, skip pulling
DEBU[0000] kernel module "ip_tables" is already loaded
DEBU[0000] kernel module "ip6_tables" is already loaded
INFO[0000] Creating lab directory: /usr/local/bugatti9/tmp/clab-srl_lab
DEBU[0000] Existing entry "Tag:       MASK (16), ID: 4294967295, Perm: rwx (7)" deleted
DEBU[0001] writing cert file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/ca/ca.pem
DEBU[0001] writing key file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/ca/ca.key
DEBU[0001] user home dir /home/schavezc found using os/user.LookupId
DEBU[0001] user home dir /home/schavezc found using os/user.LookupId
DEBU[0001] /home/schavezc/.ssh/authorized_keys found, adding it to the list of files to get public keys from
DEBU[0001] Loaded public key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC8WlbqK5AF5tOd9S5lSfXxD1NIK4KhbltWJs9D9JGIFQjsUaZwR813JIIv+uwiOYR7YmZr29RezTTlWZ82xmj0V3Ap+PB5+2SiM0aV2kyeGpwIO4SZQMq7sWO3gz0MlM5HJPYc9QppG012n/O/bgl29TSBmlMqnIXzf3oPNxf3aJkYqE7p+9Z30+LG8FGd+l8hYR7jvPJ9BZxluOI5xV/wI+y0Dsx5V8OBblhQluo4bthVOr5E/N0hWkJD3wYZWgUgcOw4JFC/2FsDhwJPUqMDFRNGOC3GW7Fk6YufBKS95AoKVCFP06NCjsy7mVX/K7L0pOsPNmvRDMBBPTppZpX0p2g0OkNY5T0F0EZhzeYv5+oi9o9l5wBRlKpAVHJmyz8aZIIoquuiVMXDFLMaEnEaEB31OuQn47aDt/q0DzB43QE6RyxXHnUvQ6tqlZehVRvZB24/+cBzHq8Q7pb4m2saimUZF09WK1cBbc8zKoJZPhAkgYjySPf9fKfBfAmPGMbqUbVgolK1afZL5QDN8Zf1BVa4QktcRZ6zRdn8jdqWSo+G7gRnVKnGq1nNedm/0eOi2xFgwXljr2+uoKr0igp8C9wRv65+DuTOetx5qwEySaxcelaal3c4EIsIar3J+E0eZJspgeJugnGvhLnU62kDUiqo/oa1gdGGVrAMhb8+Nw== schavez@bugatti9
DEBU[0001] Loaded public key ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEhohE89FYbVjan9Uh19vXPQzdyDUgJSlY9D6Y4BW6W1 schavezc
DEBU[0001] SSH_AUTH_SOCK not set, skipping pubkey fetching
DEBU[0001] Number of Node workers: 2, Number of Link workers: 2
DEBU[0001] Dependencies:
srl1 -> [  ]
srl2 -> [  ]
DEBU[0001] - cycle check round 1 -
srl1 <- [  ]
srl2 <- [  ]
DEBU[0001] node creation graph is successfully validated as being acyclic
DEBU[0001] Stage Change: Enter Wait -> srl2 - create
DEBU[0001] Stage Change: Enter Go -> srl2 - create
DEBU[0001] Worker 0 received node: &{ShortName:srl2 LongName:clab-srl_lab-srl2 Fqdn:srl2.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 Index:1 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:0xc000ae6520 ResConfig: NodeType:ixr6e Position: License:/usr/local/bugatti9/tmp/license.key Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[net.ipv4.ip_forward:0 net.ipv6.conf.all.accept_dad:0 net.ipv6.conf.all.autoconf:0 net.ipv6.conf.all.disable_ipv6:0 net.ipv6.conf.default.accept_dad:0 net.ipv6.conf.default.autoconf:0] User:0:0 Entrypoint: Cmd:sudo bash -c 'touch /.dockerenv && /opt/srlinux/bin/sr_linux' Exec:[] Env:map[CLAB_INTFS:1 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:nokia_srlinux CLAB_LABEL_CLAB_NODE_LAB_DIR:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 CLAB_LABEL_CLAB_NODE_NAME:srl2 CLAB_LABEL_CLAB_NODE_TYPE:ixr6e CLAB_LABEL_CLAB_TOPO_FILE:/usr/local/bugatti9/tmp/clab.yaml CLAB_LABEL_CONTAINERLAB:srl_lab SRLINUX:1] Binds:[/usr/local/bugatti9/tmp/clab-srl_lab/srl2/license.key:/opt/srlinux/etc/license.key:ro /usr/local/bugatti9/tmp/clab-srl_lab/srl2/config:/etc/opt/srlinux/:rw /usr/local/bugatti9/tmp/clab-srl_lab/srl2/topology.yml:/tmp/topology.yml:ro] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc001182ed0 Healthcheck:<nil> NSPath:/proc/0/ns/net Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:br-7dec063fb80a clab-node-group: clab-node-kind:nokia_srlinux clab-node-lab-dir:/usr/local/bugatti9/tmp/clab-srl_lab/srl2 clab-node-name:srl2 clab-node-type:ixr6e clab-topo-file:/usr/local/bugatti9/tmp/clab.yaml containerlab:srl_lab] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc001182f00 DNS:0xc0009ee640 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0001] Stage Change: Enter Wait -> srl1 - create
DEBU[0001] Stage Change: Enter Go -> srl1 - create
DEBU[0001] Creating directory structure for SRL container: srl2
DEBU[0001] Worker 1 received node: &{ShortName:srl1 LongName:clab-srl_lab-srl1 Fqdn:srl1.srl_lab.io LabDir:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 Index:0 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false ResStartupConfig: Config:0xc000ae64a8 ResConfig: NodeType:ixrd2 Position: License:/usr/local/bugatti9/tmp/license.key Image:harbor.ncse.io/library/srlinux:23.10.2 ImagePullPolicy:IfNotPresent Sysctls:map[net.ipv4.ip_forward:0 net.ipv6.conf.all.accept_dad:0 net.ipv6.conf.all.autoconf:0 net.ipv6.conf.all.disable_ipv6:0 net.ipv6.conf.default.accept_dad:0 net.ipv6.conf.default.autoconf:0] User:0:0 Entrypoint: Cmd:sudo bash -c 'touch /.dockerenv && /opt/srlinux/bin/sr_linux' Exec:[] Env:map[CLAB_INTFS:1 CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:nokia_srlinux CLAB_LABEL_CLAB_NODE_LAB_DIR:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 CLAB_LABEL_CLAB_NODE_NAME:srl1 CLAB_LABEL_CLAB_NODE_TYPE:ixrd2 CLAB_LABEL_CLAB_TOPO_FILE:/usr/local/bugatti9/tmp/clab.yaml CLAB_LABEL_CONTAINERLAB:srl_lab SRLINUX:1] Binds:[/usr/local/bugatti9/tmp/clab-srl_lab/srl1/license.key:/opt/srlinux/etc/license.key:ro /usr/local/bugatti9/tmp/clab-srl_lab/srl1/config:/etc/opt/srlinux/:rw /usr/local/bugatti9/tmp/clab-srl_lab/srl1/topology.yml:/tmp/topology.yml:ro] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc0011174d0 Healthcheck:<nil> NSPath:/proc/322501/ns/net Publish:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:br-7dec063fb80a clab-node-group: clab-node-kind:nokia_srlinux clab-node-lab-dir:/usr/local/bugatti9/tmp/clab-srl_lab/srl1 clab-node-name:srl1 clab-node-type:ixrd2 clab-topo-file:/usr/local/bugatti9/tmp/clab.yaml containerlab:srl_lab] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc001117500 DNS:0xc0009ee050 IsRootNamespaceBased:false SkipUniquenessCheck:false}
DEBU[0001] Creating directory structure for SRL container: srl1
DEBU[0001] CopyFile src /usr/local/bugatti9/tmp/license.key -> dst /usr/local/bugatti9/tmp/clab-srl_lab/srl1/license.key succeeded
DEBU[0001] CopyFile src /usr/local/bugatti9/tmp/license.key -> dst /usr/local/bugatti9/tmp/clab-srl_lab/srl2/license.key succeeded
DEBU[0001] {1a:08:01:00:00:00}/usr/local/bugatti9/tmp/clab-srl_lab/srl2/topology.yml
DEBU[0001] {1a:80:00:00:00:00}/usr/local/bugatti9/tmp/clab-srl_lab/srl1/topology.yml
DEBU[0001] configuration template for node srl2 is empty, skipping startup config file generation
INFO[0001] Creating container: "srl2"
DEBU[0001] configuration template for node srl1 is empty, skipping startup config file generation
INFO[0001] Creating container: "srl1"
DEBU[0001] Container "srl2" create response: {ID:38e2568fb51919e7fc1260b178f98fdcb0c8d87df46e37418775d7bdb4c707c3 Warnings:[]}
DEBU[0001] Start container: "clab-srl_lab-srl2"
DEBU[0001] Container "srl1" create response: {ID:82d12a151ad9574c6c638b9d09e70fbf5acb54c5b2e331add623d5947c71b10e Warnings:[]}
DEBU[0001] Start container: "clab-srl_lab-srl1"
DEBU[0001] Container started: "clab-srl_lab-srl2"
DEBU[0001] Filter key: name, filter value: ^clab-srl_lab-srl2$
DEBU[0001] StateChange: Done -> srl2 - create
DEBU[0001] Stage Change: Enter Wait -> srl2 - create-links
DEBU[0001] Stage Change: Enter Go -> srl2 - create-links
ERRO[0001] stage "srl2" not found
DEBU[0001] StateChange: Done -> srl2 - create-links
DEBU[0001] Stage Change: Enter Wait -> srl2 - configure
DEBU[0001] Stage Change: Enter Go -> srl2 - configure
INFO[0001] Running postdeploy actions for Nokia SR Linux 'srl2' node
DEBU[0001] creating node certificate for srl2
DEBU[0001] Container started: "clab-srl_lab-srl1"
DEBU[0001] Filter key: name, filter value: ^clab-srl_lab-srl1$
DEBU[0001] StateChange: Done -> srl1 - create
DEBU[0001] Stage Change: Enter Wait -> srl1 - create-links
DEBU[0001] Stage Change: Enter Go -> srl1 - create-links
INFO[0001] Creating link: srl1:e1-1-1 <--> srl2:e1-1-1
ERRO[0001] stage "srl1" not found
DEBU[0001] StateChange: Done -> srl1 - create-links
DEBU[0001] Stage Change: Enter Wait -> srl1 - configure
DEBU[0001] Stage Change: Enter Go -> srl1 - configure
INFO[0001] Running postdeploy actions for Nokia SR Linux 'srl1' node
DEBU[0001] creating node certificate for srl1
DEBU[0001] writing cert file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/srl2/srl2.pem
DEBU[0001] writing key file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/srl2/srl2.key
DEBU[0001] Method GetHostsPath was called with a resulting path "/var/lib/docker/containers/38e2568fb51919e7fc1260b178f98fdcb0c8d87df46e37418775d7bdb4c707c3/hosts"
DEBU[0001] Waiting for SR Linux node "srl2" to boot...
DEBU[0001] /clab-srl_lab-srl2 exec created clab-srl_lab-srl2
DEBU[0001] /clab-srl_lab-srl2 exec attached clab-srl_lab-srl2
DEBU[0002] writing cert file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/srl1/srl1.pem
DEBU[0002] writing key file to /usr/local/bugatti9/tmp/clab-srl_lab/.tls/srl1/srl1.key
DEBU[0002] Method GetHostsPath was called with a resulting path "/var/lib/docker/containers/82d12a151ad9574c6c638b9d09e70fbf5acb54c5b2e331add623d5947c71b10e/hosts"
DEBU[0002] Waiting for SR Linux node "srl1" to boot...
DEBU[0002] /clab-srl_lab-srl1 exec created clab-srl_lab-srl1
DEBU[0002] /clab-srl_lab-srl1 exec attached clab-srl_lab-srl1
DEBU[0002] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:48:06.089 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0002] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:48:06.300 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0003] /clab-srl_lab-srl2 exec created clab-srl_lab-srl2
DEBU[0003] /clab-srl_lab-srl2 exec attached clab-srl_lab-srl2
DEBU[0003] /clab-srl_lab-srl1 exec created clab-srl_lab-srl1
DEBU[0003] /clab-srl_lab-srl1 exec attached clab-srl_lab-srl1
DEBU[0004] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:48:07.651 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0004] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:48:07.845 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0005] /clab-srl_lab-srl2 exec created clab-srl_lab-srl2
DEBU[0005] /clab-srl_lab-srl2 exec attached clab-srl_lab-srl2
DEBU[0005] /clab-srl_lab-srl1 exec created clab-srl_lab-srl1
DEBU[0005] /clab-srl_lab-srl1 exec attached clab-srl_lab-srl1
DEBU[0005] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "18:48:09.139 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
DEBU[0005] mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "Parsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat']\n"
ERRO[0006] failed to create exec in container /clab-srl_lab-srl2: Error response from daemon: container 38e2568fb51919e7fc1260b178f98fdcb0c8d87df46e37418775d7bdb4c707c3 is not running
ERRO[0006] clab-srl_lab-srl2: failed to execute cmd: "/opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running" with error Error response from daemon: container 38e2568fb51919e7fc1260b178f98fdcb0c8d87df46e37418775d7bdb4c707c3 is not running
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x25eee1b]

goroutine 118 [running]:
github.com/srl-labs/containerlab/clab/exec.(*ExecResult).GetReturnCode(...)
	github.com/srl-labs/containerlab/clab/exec/exec.go:141
github.com/srl-labs/containerlab/nodes/srl.(*srl).Ready(0xc001135900, {0x36cf0c8?, 0xc0009ef810?})
	github.com/srl-labs/containerlab/nodes/srl/srl.go:333 +0x1db
github.com/srl-labs/containerlab/nodes/srl.(*srl).PostDeploy(0xc001135900, {0x36cf0c8, 0xc0009ef810}, 0xc000ae6508)
	github.com/srl-labs/containerlab/nodes/srl/srl.go:262 +0x25a
github.com/srl-labs/containerlab/clab.(*CLab).scheduleNodes.func1(0x0?, 0xc001164420, 0x0?, {0x0?, 0x0?})
	github.com/srl-labs/containerlab/clab/clab.go:589 +0x899
created by github.com/srl-labs/containerlab/clab.(*CLab).scheduleNodes
	github.com/srl-labs/containerlab/clab/clab.go:660 +0x177

[sacckth]

Issue seems to be related to the SRL image version... Have a look and let me know is this is something worth investigating clabwise.. Otherwise I'm ok with closing this

[hellt]

Hi
yes, this is not a clab issue, hence closing.