idb_server crashes on SR Linux

[alistairking]

Hey!
First, thanks for this excellent project.

I've been playing around a bit with container lab and have managed to get cEOS, FRR, and vMX routers up, but for some reason I can't get SRL to boot.
I've searched through existing issues but haven't managed to find anything similar, but apologies if I missed something.
I've tried both 25.3.1 and 24.10.3.

Relevant (or at least, what I think are relevant) logs are pasted below.

Thanks for your help!

root@spine01:/# head -50 /var/log/messages
Mar 22 00:01:25 spine01 rsyslogd: rsyslogd's groupid changed to 4
Mar 22 00:01:25 spine01 rsyslogd: rsyslogd's userid changed to 999
Mar 22 00:01:25 spine01 rsyslogd: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="1428" x-info="https://www.rsyslog.com"] start
Mar 22 00:01:26 spine01 rsyslogd: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="1428" x-info="https://www.rsyslog.com"] exiting on signal 15.
Mar 22 00:01:26 spine01 sr_cli:Error in accessing file /var/run/srlinux/devices/hw_details.json
Mar 22 00:01:26 spine01 sr_app_mgr: app|1963|1963|00088|N: Starting application supportd
Mar 22 00:01:26 spine01 rsyslogd: rsyslogd's groupid changed to 4
Mar 22 00:01:26 spine01 rsyslogd: rsyslogd's userid changed to 999
Mar 22 00:01:26 spine01 rsyslogd: [origin software="rsyslogd" swVersion="8.2302.0" x-pid="1934" x-info="https://www.rsyslog.com"] start
Mar 22 00:01:27 spine01 sr_app_mgr: app|1963|1963|00095|N: Starting application dev_mgr
Mar 22 00:01:27 spine01 sr_app_mgr: app|1963|1963|00134|N: Starting application idb_server
Mar 22 00:01:27 spine01 sr_idb_server: debug|2112|2112|00018|C: {part#1/2} common    hasync_ipc.c:1157                SRL_FAIL_MSG  Termination handler
========= SRL_FAIL_MSG in program: sr_idb_server =========
=== At: /builds/sr/srlinux/srlutil/idb/ipc/hasync_ipc.c:1157
=== Failed to bind to unix socket: /opt/srlinux/var/run/sr_idb_server:50010

 0x7fa348b26992, /opt/srlinux/lib/liblib_srlutil_idb__ipc.so : ()+0xa992
 0x7fa348b27fca, /opt/srlinux/lib/liblib_srlutil_idb__ipc.so : HasyncInitServer()+0x9a
 0x7fa349730566, /opt/srlinux/lib/liblib_srlutil_idb__client.so : srlinux::IdbServer::StartIpcServer()+0x256
 0x7fa34973866d, /opt/srlinux/lib/liblib_srlutil_idb__client.so : srlinux::IdbServer::IdbServer(ELoopHandle*, srlinux::IdbServerApiOptions)+0xb2d
 0x7fa3497390d5, /opt/srlinux/lib/liblib_srlutil_idb__client.so : srlinux::IdbServerApi::IdbServerCreate(ELoopHandle*, srlinux::IdbServerApiOptions)+0xc5
 0x556653a99701, ./sr_idb_server : main()+0xaf1
 0x7fa345af524a, /lib/x86_64-linux-gnu/libc.so.6
Mar 22 00:01:27 spine01 sr_idb_server: debug|2112|2112|00018|C: {part#2/2} : ()+0x2724a
 0x7fa345af5305, /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main()+0x85
 0x556653a96781, ./sr_idb_server : _start()+0x21
==============================
=== Program: sr_idb_server
/builds/sr/srlinux/srlutil/idb/ipc/hasync_ipc.c:1157 | Failed to bind to unix socket: /opt/srlinux/var/run/sr_idb_server:50010

[hellt]

Hi! What os is that and how many cpu ram is there?

[alistairking]

This is a bare metal 80 core / 512 GB RAM server running Debian Bullseye (kernel version 6.6.71)

[hellt]

This is super weird
we have been running SR Linux on all kinds of HW/OS/Kernel combos, and never saw this issue.
I will check if someone internally saw anything like this.

In the meantime can you paste the full log of the following command:

clab dep -d -c -t srlinux.dev/clab-srl

[alistairking]

Here is the output. I had to slightly modify the lab definition to disable the IPv6 management network because it didn't play nicely with iptables on this host.

$ clab dep -d -c -t clab.yml
21:49:25 DEBU Changed running UIDs to UID: 1094 GID: 1094
21:49:25 DEBU Dropped root privileges
21:49:25 DEBU Group membership check passed
21:49:25 DEBU Changed running UIDs to UID: 0 GID: 1094
21:49:25 DEBU Obtained root privileges
21:49:25 INFO Containerlab started version=0.66.0
21:49:25 DEBU template variables: <nil>
21:49:25 DEBU topology:
name: srl
# node name will be srl unless overridden by CLAB_PREFIX
prefix: ${CLAB_PREFIX:=""}

topology:
  nodes:
    srl:
      kind: nokia_srlinux
      image: ${SRL_IMAGE:=ghcr.io/nokia/srlinux}:${SRL_VERSION:=latest}
      type: ${SRL_TYPE:=ixrd3l}

mgmt:
  network: mgmt                # management network name
  ipv4-subnet: 172.100.100.0/24       # ipv4 range
  ipv6-subnet:


21:49:25 DEBU method initMgmtNetwork was called mgmt params &{Network:mgmt Bridge: IPv4Subnet:172.100.100.0/24 IPv4Gw: IPv4Range: IPv6Subnet: IPv6Gw: IPv6Range: MTU:0 ExternalAccess:<nil>}
21:49:25 DEBU New mgmt params are &{Network:mgmt Bridge: IPv4Subnet:172.100.100.0/24 IPv4Gw: IPv4Range: IPv6Subnet: IPv6Gw: IPv6Range: MTU:0 ExternalAccess:0xc0003b8fbc}
21:49:25 DEBU env runtime var value is
21:49:25 DEBU Running runtime.Init with params &{Timeout:2m0s GracefulShutdown:false Debug:false KeepMgmtNet:false VerifyLinkParams:<nil>} and &{Network:mgmt Bridge: IPv4Subnet:172.100.100.0/24 IPv4Gw: IPv4Range: IPv6Subnet: IPv6Gw: IPv6Range: MTU:0 ExternalAccess:0xc0003b8fbc}
21:49:25 DEBU Runtime: Docker
21:49:25 DEBU initialized a runtime with params &{config:{Timeout:120000000000 GracefulShutdown:false Debug:false KeepMgmtNet:false VerifyLinkParams:0xc00039425f} Client:0xc0005b2d20 mgmt:0xc000666000}
21:49:25 INFO Parsing & checking topology file=clab.yml
21:49:25 DEBU node config: &{ShortName:srl LongName:srl Fqdn:srl.srl.io LabDir:/home/alistair/test/clab-srl/srl Index:0 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false RestartPolicy: ResStartupConfig: Config:<nil> ResConfig: NodeType:ixrd3l Position: License: Image:ghcr.io/nokia/srlinux:latest ImagePullPolicy:IfNotPresent Sysctls:map[] User: Entrypoint: Cmd: Exec:[] Env:map[NO_PROXY:localhost,127.0.0.1,::1,*.local,172.100.100.0/24,srl no_proxy:localhost,127.0.0.1,::1,*.local,172.100.100.0/24,srl] Binds:[] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc0005530e0 Healthcheck:<nil> Aliases:[] ExtraHosts:[] Labels:map[] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc000553140 DNS:0xc00039c0f0 IsRootNamespaceBased:false SkipUniquenessCheck:false}
21:49:25 DEBU Error opening host DNS config run/systemd/resolve/resolv.conf: open run/systemd/resolve/resolv.conf: no such file or directory
21:49:25 DEBU Env: CLAB_VERSION_CHECK=
21:49:25 DEBU Number of Node workers: 80
21:49:25 DEBU lab Conf: &{Name:srl Prefix:0xc000782eb0 Mgmt:0xc000666000 Settings:<nil> Topology:0xc0006640c0 Debug:false}
21:49:25 DEBU Filter key: name, filter value: ^srl$
21:49:25 INFO Removing directory path=/home/alistair/test/clab-srl
21:49:25 DEBU Checking if docker network "mgmt" exists
21:49:25 DEBU Network does not exist name=mgmt
21:49:25 INFO Creating docker network name=mgmt IPv4 subnet=172.100.100.0/24 IPv6 subnet="" MTU=0
21:49:26 DEBU Docker network "mgmt", bridge name "br-58256d68c814"
21:49:26 DEBU Disable RPF check on the docker host
21:49:26 DEBU Enable LLDP on the linux bridge br-58256d68c814
21:49:26 DEBU Disabling TX checksum offloading for the br-58256d68c814 bridge interface...
21:49:26 DEBU nftables does not seem to be in use, no DOCKER-USER chain found.
21:49:26 DEBU using ip_tables as the firewall interface
21:49:26 WARN iptables check error: exec: "iptables": executable file not found in $PATH. Output:
21:49:26 WARN iptables check error: exec: "ip6tables": executable file not found in $PATH. Output:
21:49:26 WARN iptables check error: exec: "iptables": executable file not found in $PATH. Output:
21:49:26 WARN iptables check error: exec: "ip6tables": executable file not found in $PATH. Output:
21:49:26 DEBU kernel version: 6.6.71

21:49:26 DEBU Looking up ghcr.io/nokia/srlinux:latest Docker image
21:49:26 DEBU SUDO_UID env var is not set, using current user home dir: /home/alistair
21:49:26 DEBU Could not read docker config: open /home/alistair/.docker/config.json: no such file or directory
21:49:26 DEBU docker config file not found
21:49:26 INFO Pulling ghcr.io/nokia/srlinux:latest Docker image
21:49:26 INFO Done pulling ghcr.io/nokia/srlinux:latest
21:49:26 DEBU kernel module "ip_tables" is already loaded
21:49:26 DEBU kernel module "ip6_tables" is already loaded
21:49:26 INFO Creating lab directory path=/home/alistair/test/clab-srl
21:49:26 INFO unable to adjust Labdir file ACLs: operation not supported
21:49:26 DEBU writing cert file to /home/alistair/test/clab-srl/.tls/ca/ca.pem
21:49:26 DEBU writing key file to /home/alistair/test/clab-srl/.tls/ca/ca.key
21:49:26 DEBU SUDO_UID env var is not set, using current user home dir: /home/alistair
21:49:26 DEBU SUDO_UID env var is not set, using current user home dir: /home/alistair
21:49:26 DEBU /home/alistair/.ssh/authorized_keys found, adding it to the list of files to get public keys from
21:49:26 DEBU Loaded public key ssh-rsa <snip>
21:49:26 DEBU extracted 6 keys from ssh-agent
21:49:26 DEBU Dependencies:
srl -> [  ]
21:49:26 DEBU - cycle check round 1 -
srl <- [  ]
21:49:26 DEBU node creation graph is successfully validated as being acyclic
21:49:26 DEBU Stage Change: Enter Wait -> srl - create
21:49:26 DEBU Stage Change: Enter Go -> srl - create
21:49:26 DEBU Worker 0 received node: &{ShortName:srl LongName:srl Fqdn:srl.srl.io LabDir:/home/alistair/test/clab-srl/srl Index:0 Group: Kind:nokia_srlinux StartupConfig: StartupDelay:0 EnforceStartupConfig:false SuppressStartupConfig:false AutoRemove:false RestartPolicy: ResStartupConfig: Config:0xc000634040 ResConfig: NodeType:ixrd3l Position: License: Image:ghcr.io/nokia/srlinux:latest ImagePullPolicy:IfNotPresent Sysctls:map[net.ipv4.ip_forward:0 net.ipv6.conf.all.accept_dad:0 net.ipv6.conf.all.autoconf:0 net.ipv6.conf.all.disable_ipv6:0 net.ipv6.conf.default.accept_dad:0 net.ipv6.conf.default.autoconf:0] User:0:0 Entrypoint: Cmd:sudo bash -c 'touch /.dockerenv && /opt/srlinux/bin/sr_linux' Exec:[] Env:map[CLAB_LABEL_CLAB_NODE_GROUP: CLAB_LABEL_CLAB_NODE_KIND:nokia_srlinux CLAB_LABEL_CLAB_NODE_LAB_DIR:/home/alistair/test/clab-srl/srl CLAB_LABEL_CLAB_NODE_LONGNAME:srl CLAB_LABEL_CLAB_NODE_NAME:srl CLAB_LABEL_CLAB_NODE_TYPE:ixrd3l CLAB_LABEL_CLAB_OWNER:alistair CLAB_LABEL_CLAB_TOPO_FILE:/home/alistair/test/clab.yml CLAB_LABEL_CONTAINERLAB:srl NO_PROXY:localhost,127.0.0.1,::1,*.local,172.100.100.0/24,srl SRLINUX:1 no_proxy:localhost,127.0.0.1,::1,*.local,172.100.100.0/24,srl] Binds:[/home/alistair/test/clab-srl/srl/config:/etc/opt/srlinux/:rw /home/alistair/test/clab-srl/srl/topology.yml:/tmp/topology.yml:ro] PortBindings:map[] ResultingPortBindings:[] PortSet:map[] NetworkMode: MgmtNet: MgmtIntf: MgmtIPv4Address: MgmtIPv4PrefixLength:0 MgmtIPv6Address: MgmtIPv6PrefixLength:0 MgmtIPv4Gateway: MgmtIPv6Gateway: MacAddress: ContainerID: TLSCert: TLSKey: TLSAnchor: Certificate:0xc0005530e0 Healthcheck:<nil> Aliases:[] ExtraHosts:[] Labels:map[clab-mgmt-net-bridge:br-58256d68c814 clab-node-group: clab-node-kind:nokia_srlinux clab-node-lab-dir:/home/alistair/test/clab-srl/srl clab-node-longname:srl clab-node-name:srl clab-node-type:ixrd3l clab-owner:alistair clab-topo-file:/home/alistair/test/clab.yml containerlab:srl] Sandbox: Kernel: Runtime: CPU:0 CPUSet: Memory: Extras:<nil> Stages:0xc000553140 DNS:0xc00039c0f0 IsRootNamespaceBased:false SkipUniquenessCheck:false}
21:49:26 DEBU Creating directory structure for SRL container: srl
21:49:26 DEBU {1a:64:00:00:00:00} /home/alistair/test/clab-srl/srl/topology.yml="missing value"
21:49:26 DEBU configuration template for node srl is empty, skipping startup config file generation
21:49:26 INFO Creating container name=srl
21:49:26 DEBU Container "srl" create response: {ID:2acfbede72cbf24fdd111743335c4fecde4a6257e77688f7c86749c461d05efa Warnings:[]}
21:49:26 DEBU Start container: "srl"
21:49:27 DEBU Container started: "srl"
21:49:27 DEBU Filter key: name, filter value: ^srl$
21:49:27 DEBU StateChange: Done -> srl - create
21:49:27 DEBU Stage Change: Enter Wait -> srl - create-links
21:49:27 DEBU Stage Change: Enter Go -> srl - create-links
21:49:27 DEBU StateChange: Done -> srl - create-links
21:49:27 DEBU Stage Change: Enter Wait -> srl - configure
21:49:27 DEBU Stage Change: Enter Go -> srl - configure
21:49:27 INFO Running postdeploy actions kind=nokia_srlinux node=srl
21:49:27 DEBU creating node certificate for srl
21:49:27 DEBU writing cert file to /home/alistair/test/clab-srl/.tls/srl/srl.pem
21:49:27 DEBU writing key file to /home/alistair/test/clab-srl/.tls/srl/srl.key
21:49:27 DEBU Method GetHostsPath was called with a resulting path "/var/lib/docker/containers/2acfbede72cbf24fdd111743335c4fecde4a6257e77688f7c86749c461d05efa/hosts"
21:49:27 DEBU Waiting for SR Linux node "srl" to boot...
21:49:27 DEBU /srl exec created srl
21:49:27 DEBU /srl exec attached srl
21:49:28 DEBU mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "21:49:28.148 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat', 'with-context', 'yang-models', '|']\n"
21:49:29 DEBU /srl exec created srl
21:49:29 DEBU /srl exec attached srl
21:49:30 DEBU mgmt_server status check failed, output:
Cmd: /opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running
ReturnCode: 1
Stderr: "21:49:29.965 [WARNING] (MainThread) Failed to contact the AAA server. Further communication will be blackholed until connection succeeds\nParsing error: Unknown token 'system'. Options are ['#', '>', '>>', 'depth', 'detail', 'flat', 'with-context', 'yang-models', '|']\n"
21:49:31 DEBU /srl exec created srl
21:49:31 DEBU /srl exec attached srl

And this error repeats.

Docker logs is similar to what I reported above

sudo docker logs srl
Sat Mar 22 21:49:27 UTC 2025: entrypoint.sh called
Sat Mar 22 21:49:27 UTC 2025: Calling boot_run script
Actual changes:
tx-checksum-ip-generic: off
tx-tcp-segmentation: off [not requested]
tx-tcp-ecn-segmentation: off [not requested]
tx-tcp-mangleid-segmentation: off [not requested]
tx-tcp6-segmentation: off [not requested]
tx-checksum-sctp: off
rx-checksum: off
net.ipv4.ip_local_port_range = 55000 65535
1000
No op
Sat Mar 22 21:49:29 UTC 2025: entrypoint.sh done, executing sudo bash -c touch /.dockerenv && /opt/srlinux/bin/sr_linux

/usr/bin/sudo --non-interactive -- /opt/srlinux/bin/sr_linux --sudo-switched

sr_app_mgr: Not starting in a named namespace, giving it the name "srbase"
sr_app_mgr: Unix domain socket directory is /opt/srlinux/var/run/
Log directory is /var/log/srlinux/stdout
  Found supportd: PID 1383
  Application supportd is running: PID 1383
  Started dev_mgr: source /etc/profile.d/sr_app_env.sh &>/dev/null; /usr/bin/bash -lc "./sr_device_mgr" >>/var/log/srlinux/stdout/dev_mgr.log 2>&1 &
  Application dev_mgr is running: PID 1957
  Started idb_server: source /etc/profile.d/sr_app_env.sh &>/dev/null; /usr/bin/bash -lc "./sr_idb_server" >>/var/log/srlinux/stdout/idb_server.log 2>&1 &
  Started idb_server: source /etc/profile.d/sr_app_env.sh &>/dev/null; /usr/bin/bash -lc "./sr_idb_server" >>/var/log/srlinux/stdout/idb_server.log 2>&1 &
  Application idb_server is running: PID 2859
  Started idb_server: source /etc/profile.d/sr_app_env.sh &>/dev/null; /usr/bin/bash -lc "./sr_idb_server" >>/var/log/srlinux/stdout/idb_server.log 2>&1 &
  Application idb_server is running: PID 3475
25-03-22 21:49:50.644 sr_app_mgr: Would reboot with reason 'AppMgr: idb_server has failed' but no-reboot is set

[hellt]

@alistairking is there any other compute node in your lab that you can try the lab on? It looks like some security hardening prevents the container to bind to the socket, but as I never saw this before since 3 years, it is hard to pinpoint the issue right away/

[alistairking]

I can try and find some other hardware to try this on.

I'm not sure if this means anything, but it seems like if I try and run sr_idb_server by hand then it seems to starts ok -- or at least it doesn't crash...

sudo docker exec -it srl bash
root@srl:/# cd opt/srlinux/bin/
root@srl:/opt/srlinux/bin# rm /opt/srlinux/var/run/dev_mgr-lock
root@srl:/opt/srlinux/bin# source /etc/profile.d/sr_app_env.sh &>/dev/null; /usr/bin/bash -lc "./sr_device_mgr" >>/var/log/srlinux/stdout/dev_mgr.log 2>&1

and logs

root@srl:/# cat /var/log/srlinux/stdout/dev_mgr.log
E0325 16:58:58.463022764    7839 chttp2_server.cc:1045]                UNKNOWN:No address added out of total 1 resolved for '0.0.0.0:50065' {created_time:"2025-03-25T16:58:58.462726022+00:00", file_line:960, file:"./src/core/ext/transport/chttp2/server/chttp2_server.cc", children:[UNKNOWN:Failed to add any wildcard listeners {created_time:"2025-03-25T16:58:58.462172246+00:00", file_line:353, file:"./src/core/lib/iomgr/tcp_server_posix.cc", children:[UNKNOWN:Unable to configure socket {file:"./src/core/lib/iomgr/tcp_server_utils_posix_common.cc", file_line:219, created_time:"2025-03-25T16:58:58.460489881+00:00", fd:50, children:[UNKNOWN:Address already in use {syscall:"bind", os_error:"Address already in use", errno:98, created_time:"2025-03-25T16:58:58.460436291+00:00", file_line:193, file:"./src/core/lib/iomgr/tcp_server_utils_posix_common.cc"}]}, UNKNOWN:Unable to configure socket {file:"./src/core/lib/iomgr/tcp_server_utils_posix_common.cc", file_line:219, created_time:"2025-03-25T16:58:58.462165094+00:00", fd:50, children:[UNKNOWN:Address already in use {file:"./src/core/lib/iomgr/tcp_server_utils_posix_common.cc", file_line:193, created_time:"2025-03-25T16:58:58.462155532+00:00", errno:98, os_error:"Address already in use", syscall:"bind"}]}]}]}

[hellt]

I believe you managed to resolve this @alistairking ?
After talking to devs it seems the issue is/was quite specific to your infra, as this is the first time something like this was brought up and there are many SR Linux users out there with different deployment models.

unfortunately, without having access to this infra from our side it is quite challenging to do effective debugging, so unless there is a chance to share an instance for tshooting I would say not much we can do to put a finger on the issue.

[alistairking]

Unfortunately I wasn't able to get it to work on our hardware and ended up moving to running the lab on a GCP VM.

I appreciate you looking into the problem.

I'll close the issue since it doesn't seem to be a container lab problem.

[hvisage]

Good day,

So, doing a containerlab I setup my own Debian docker setup, and ran into this issue... debugging the ai got me fixed with this explanation to fix the ZFS ACL settings

SR Linux on Containerlab fails on Docker+ZFS with acltype=off

TL;DR

If you run Nokia SR Linux under Containerlab and the host's Docker is backed by ZFS
with acltype=off (the OpenZFS default), sr_idb_server crashes during boot with:

Failed to bind to unix socket: /opt/srlinux/var/run/sr_idb_server:50010

Fix: set acltype=posix on the Docker ZFS dataset, wipe, and re-pull the image.

sudo systemctl stop docker docker.socket
sudo zfs set acltype=posix aclinherit=passthrough <docker-dataset>
sudo systemctl start docker
docker system prune -af --volumes
docker pull ghcr.io/nokia/srlinux:<tag>
clab deploy -t <topology>.clab.yaml

---

Symptoms

clab deploy times out waiting for the SR Linux node to boot:

ERRO failed to run postdeploy task for node leaf1:
     timed out waiting for SR Linux node leaf1 to boot: <nil>

docker ps shows the container as Up, but nothing works inside it.
/var/log/srlinux/stdout/idb_server.log contains:

SRL_FAIL_MSG in program: sr_idb_server
=== At: /builds/sr/srlinux/srlutil/idb/ipc/hasync_ipc.c:1165
=== Failed to bind to unix socket: /opt/srlinux/var/run/sr_idb_server:50010

sr_idb_server is the state store the rest of the SR Linux app stack depends on.
When it dies, sr_cli errors out, management interfaces don't come up, and
containerlab's post-deploy health check times out.

Confirming the environment

# Is Docker on ZFS?
docker info | grep -iE 'storage driver|backing filesystem'

# Check the dataset's ACL setting:
sudo zfs get xattr,acltype,aclinherit <docker-dataset>

Broken configuration looks like:

NAME           PROPERTY    VALUE          SOURCE
Docker/varlib  xattr       sa             default
Docker/varlib  acltype     off            default   <-- this
Docker/varlib  aclinherit  restricted     default

Inside a failing container:

docker exec <clab-node> ls -la /opt/srlinux/var/run/

Broken — plain drwxr-xr-x root:root, no + ACL marker, sr_idb_server:50010 missing:

drwxr-xr-x 2 root    root    7 .
-rw-rw---- 1 srlinux srlinux 0 app_mgr-lock
srwxrwxrw- 1 srlinux srlinux 0 dev_mgr:50011
srwxrwxrw- 1 srlinux srlinux 0 vxdp_launcher

Working — drwxrwxr-x+ with ACL marker, full socket set:

drwxrwxr-x+ 2 root    root    4096 .
-rw-rw----+ 1 srlinux srlinux ...
srwxrwxrw-+ 1 srlinux srlinux ... dev_mgr:50011
srwxrwxrw-+ 1 srlinux srlinux ... sr_idb_server:50010
... (plus sr_mgmt_server, sr_netconf_mgr, sr_grpc_server_*, etc.)

Root cause

The SR Linux image ships a POSIX ACL on /opt/srlinux/var/run/ granting the
srlinux group write access. ZFS with acltype=off silently discards POSIX ACLs
during image unpack, leaving the directory as plain mode 755 owned by root.

SR Linux apps run as the unprivileged srlinux user (uid 1002). Most of the
sr_* binaries have cap_dac_override in their file capabilities and can bind
sockets in the root-owned directory regardless. Check with getcap:

docker exec <clab-node> bash -c '
  for b in sr_idb_server sr_device_mgr sr_app_mgr sr_mgmt_server; do
    echo "--- $b ---"
    getcap /opt/srlinux/bin/$b
  done'

Typical output:

--- sr_idb_server ---
/opt/srlinux/bin/sr_idb_server cap_sys_resource=eip
--- sr_device_mgr ---
/opt/srlinux/bin/sr_device_mgr cap_chown,cap_dac_override,cap_fowner,...=eip
--- sr_app_mgr ---
/opt/srlinux/bin/sr_app_mgr cap_chown,cap_dac_override,cap_fowner,...=eip
--- sr_mgmt_server ---
/opt/srlinux/bin/sr_mgmt_server cap_sys_resource=eip

sr_idb_server only has cap_sys_resource — no DAC override. When the ACL is
missing, its bind() returns EACCES and it exits. sr_device_mgr succeeds
because it has DAC override, which is why the failure pattern looks asymmetric
— some sockets are created, others aren't.

Fix

# Stop Docker and any running labs first
clab destroy -t <topology>.clab.yaml --cleanup
sudo systemctl stop docker docker.socket

# Set the required ZFS properties on the Docker dataset
sudo zfs set acltype=posix aclinherit=passthrough <docker-dataset>

# Verify
sudo zfs get xattr,acltype,aclinherit <docker-dataset>
# Should show: xattr=sa, acltype=posix, aclinherit=passthrough

# Existing datasets and image layers don't retroactively gain ACLs.
# Wipe cached images and re-pull so new layers inherit the ACL correctly.
sudo systemctl start docker
docker system prune -af --volumes
docker pull ghcr.io/nokia/srlinux:<tag>

clab deploy -t <topology>.clab.yaml

Nuclear option if docker system prune doesn't clear everything:

sudo systemctl stop docker docker.socket
sudo zfs destroy -r <docker-dataset>
sudo zfs create -o acltype=posix -o aclinherit=passthrough -o xattr=sa \
                -o mountpoint=/var/lib/docker <docker-dataset>
sudo systemctl start docker

Verification

After redeploying, the run directory should show ACLs and the full socket set:

docker exec <clab-node> ls -la /opt/srlinux/var/run/

drwxrwxr-x+ 2 root    root    28 .
srwxrwxrw-+ 1 srlinux srlinux  0 dev_mgr:50011
srwxrwxrw-+ 1 srlinux srlinux  0 sr_idb_server:50010    <-- present
srwxrwxrwx+ 1 srlinux srlinux  0 sr_mgmt_server:50051
...

Why docker run sometimes appears to work

Running the SR Linux image directly with docker run --privileged ... sudo bash -c '... sr_linux' can appear to succeed in broken ACL environments — parent
processes may stay as root for longer and create the socket before the user
switch. Containerlab's startup exposes the missing ACL more reliably.

Reproduction environment

Failed on:

• Debian 13 (trixie) cloud image, kernel 6.16.12+deb13-cloud-amd64, QEMU/KVM guest
• Docker 29.4.0, storage-driver=zfs, backing filesystem=zfs
• ZFS: xattr=sa, acltype=off (default), aclinherit=restricted (default)
• Containerlab 0.74.3
• Failed identically on ghcr.io/nokia/srlinux:25.10 and ghcr.io/nokia/srlinux:26.3

Worked on:

• Debian 13 (trixie), kernel 6.12.73+deb13-cloud-amd64, QEMU/KVM
• Docker 29.4.0, storage-driver=overlay2, backing filesystem=extfs
• Containerlab 0.74.3
• SR Linux 25.10

Related

• srl-labs/containerlab #2518 — identical symptom, closed as "not planned" without resolution published.
• Docker ZFS storage driver docs — recommends xattr=sa but does not mention acltype=posix.

Suggested upstream improvements

For containerlab: preflight check that warns when Docker is on ZFS with
acltype != posix, and a line in the Nokia SR Linux kind documentation about
the requirement.

For the SR Linux image: either (a) have the entrypoint chmod 775 /opt/srlinux/var/run/ defensively so the directory works without relying on
ACLs, or (b) add cap_dac_override to sr_idb_server's file capabilities so
it behaves like the other sr_* managers.