Prebuilt images fail to start on Ubuntu 20.04 focal

[truelai]

$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.2 LTS"
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

$ docker-compose pull
Pulling db         ... done
Pulling es         ... done
Pulling rabbit     ... done
Pulling redis      ... done
Pulling serviceapi ... done
Pulling webapi     ... done
Pulling frontend   ... done
Pulling pipeline0  ... done
Pulling mount-1    ... done

$ docker-compose up -d
Creating network "ambar_internal_network" with the default driver
Creating ambar_rabbit_1 ... error
Creating ambar_es_1     ... 
Creating ambar_redis_1  ... 
Creating ambar_es_1     ... error

ERROR: for ambar_rabbit_1  Cannot start service rabbit: ttrpc: closed: unknown
Creating ambar_redis_1  ... error
ERROR: for ambar_es_1  Cannot start service es: ttrpc: closed: unknown

Creating ambar_db_1     ... error

ERROR: for ambar_db_1  Cannot start service db: ttrpc: closed: unknown

ERROR: for rabbit  Cannot start service rabbit: ttrpc: closed: unknown

ERROR: for es  Cannot start service es: ttrpc: closed: unknown

ERROR: for redis  Cannot start service redis: ttrpc: closed: unknown

ERROR: for db  Cannot start service db: ttrpc: closed: unknown
ERROR: Encountered errors while bringing up the project.

[truelai]

From syslog

Jun 1 18:10:16 docker systemd-networkd[180]: rtnl: received neighbor for link '30' we don't know about, ignoring.
Jun 1 18:10:16 docker systemd-networkd[180]: message repeated 3 times: [ rtnl: received neighbor for link '30' we don't know about, ignoring.]
Jun 1 18:10:16 docker networkd-dispatcher[210]: WARNING:Unknown index 30 seen, reloading interface list
Jun 1 18:10:16 docker systemd-networkd[180]: br-ce076d1533de: Link UP
Jun 1 18:10:16 docker systemd-networkd[180]: vethd69ae2b: Link UP
Jun 1 18:10:16 docker networkd-dispatcher[210]: WARNING:Unknown index 31 seen, reloading interface list
Jun 1 18:10:16 docker kernel: [ 2610.230853] br-ce076d1533de: port 1(vethd69ae2b) entered blocking state
Jun 1 18:10:16 docker kernel: [ 2610.230901] device vethd69ae2b entered promiscuous mode
Jun 1 18:10:16 docker kernel: [ 2610.230974] br-ce076d1533de: port 1(vethd69ae2b) entered forwarding state
Jun 1 18:10:16 docker networkd-dispatcher[210]: WARNING:Unknown index 33 seen, reloading interface list
Jun 1 18:10:16 docker systemd-networkd[180]: veth60ddb1f: Link UP
Jun 1 18:10:16 docker systemd-networkd[180]: br-ce076d1533de: Gained carrier
Jun 1 18:10:16 docker kernel: [ 2610.281652] br-ce076d1533de: port 2(veth60ddb1f) entered disabled state
Jun 1 18:10:16 docker kernel: [ 2610.281786] br-ce076d1533de: port 2(veth60ddb1f) entered blocking state
Jun 1 18:10:16 docker kernel: [ 2610.281815] IPv6: ADDRCONF(NETDEV_CHANGE): br-ce076d1533de: link becomes ready
Jun 1 18:10:17 docker networkd-dispatcher[210]: WARNING:Unknown index 35 seen, reloading interface list
Jun 1 18:10:17 docker systemd-networkd[180]: veth2d527f5: Link UP
Jun 1 18:10:17 docker kernel: [ 2610.436035] br-ce076d1533de: port 4(vethfedb981) entered forwarding state
Jun 1 18:10:17 docker networkd-dispatcher[210]: WARNING:Unknown index 37 seen, reloading interface list
Jun 1 18:10:17 docker systemd-networkd[180]: vethfedb981: Link UP
Jun 1 18:10:17 docker systemd-networkd[180]: vethd69ae2b: Link DOWN
Jun 1 18:10:17 docker kernel: [ 2610.558268] device vethd69ae2b left promiscuous mode
Jun 1 18:10:17 docker systemd-networkd[180]: rtnl: received neighbor for link '32' we don't know about, ignoring.
Jun 1 18:10:17 docker systemd-networkd[180]: rtnl: received neighbor for link '32' we don't know about, ignoring.
Jun 1 18:10:17 docker systemd-networkd[180]: veth60ddb1f: Link DOWN
Jun 1 18:10:17 docker systemd-networkd[180]: rtnl: received neighbor for link '34' we don't know about, ignoring.
Jun 1 18:10:17 docker systemd-networkd[180]: rtnl: received neighbor for link '34' we don't know about, ignoring.
Jun 1 18:10:17 docker docker.dockerd[214]: time="2021-06-01T18:10:17.435996332Z" level=error msg="48597d20c27a2941a683f67628b799264dfb4e06513f182a61a24eb4da26e91b cleanup: failed to delete container from containerd: no such container"
Jun 1 18:10:17 docker docker.dockerd[214]: time="2021-06-01T18:10:17.436042592Z" level=error msg="Handler for POST /v1.24/containers/48597d20c27a2941a683f67628b799264dfb4e06513f182a61a24eb4da26e91b/start returned error: error while creating mount source path '/data/rabbit': mkdir /data: read-only file system"
Jun 1 18:10:17 docker systemd-networkd[180]: vethfedb981: Link DOWN
Jun 1 18:10:17 docker systemd-networkd[180]: rtnl: received neighbor for link '38' we don't know about, ignoring.
Jun 1 18:10:17 docker kernel: [ 2610.796397] br-ce076d1533de: port 3(veth2d527f5) entered disabled state
Jun 1 18:10:17 docker systemd-networkd[180]: rtnl: received neighbor for link '38' we don't know about, ignoring.
Jun 1 18:10:17 docker docker.dockerd[214]: time="2021-06-01T18:10:17.513995408Z" level=error msg="ec8f7714d219590cbca2b2bd42113c310952fab475eef33504d71f0507318741 cleanup: failed to delete container from containerd: no such container"
Jun 1 18:10:17 docker docker.dockerd[214]: time="2021-06-01T18:10:17.514038028Z" level=error msg="Handler for POST /v1.24/containers/ec8f7714d219590cbca2b2bd42113c310952fab475eef33504d71f0507318741/start returned error: error while creating mount source path '/data/es': mkdir /data: read-only file system"
Jun 1 18:10:17 docker docker.dockerd[214]: time="2021-06-01T18:10:17.550981625Z" level=error msg="d8be67817db5341d983f0c75bfdc7be0d0a91cc98dcb06c0550d8b26c900566b cleanup: failed to delete container from containerd: no such container"
Jun 1 18:10:17 docker docker.dockerd[214]: time="2021-06-01T18:10:17.551022215Z" level=error msg="Handler for POST /v1.24/containers/d8be67817db5341d983f0c75bfdc7be0d0a91cc98dcb06c0550d8b26c900566b/start returned error: error while creating mount source path '/data/db': mkdir /data: read-only file system"
Jun 1 18:10:17 docker kernel: [ 2610.956429] eth0: renamed from veth9575ab0
Jun 1 18:10:17 docker systemd-networkd[180]: veth2d527f5: Gained carrier
Jun 1 18:10:17 docker systemd-networkd[180]: veth2d527f5: Lost carrier
Jun 1 18:10:17 docker kernel: [ 2611.067401] br-ce076d1533de: port 3(veth2d527f5) entered disabled state
Jun 1 18:10:17 docker kernel: [ 2611.067462] veth9575ab0: renamed from eth0
Jun 1 18:10:17 docker networkd-dispatcher[210]: WARNING:Unknown index 35 seen, reloading interface list
Jun 1 18:10:17 docker systemd-networkd[180]: veth2d527f5: Link DOWN
Jun 1 18:10:17 docker systemd-networkd[180]: rtnl: received neighbor for link '36' we don't know about, ignoring.
Jun 1 18:10:17 docker systemd-networkd[180]: rtnl: received neighbor for link '36' we don't know about, ignoring.
Jun 1 18:10:17 docker docker.dockerd[214]: time="2021-06-01T18:10:17.925446713Z" level=error msg="c16d4cbbee7899ff568e2e2835790e7449657d4394388528621b993ed6ad1a40 cleanup: failed to delete container from containerd: no such container"
Jun 1 18:10:17 docker docker.dockerd[214]: time="2021-06-01T18:10:17.925484063Z" level=error msg="Handler for POST /v1.24/containers/c16d4cbbee7899ff568e2e2835790e7449657d4394388528621b993ed6ad1a40/start returned error: cgroups: cgroup mountpoint does not exist: unknown"
Jun 1 18:10:18 docker systemd-networkd[180]: br-ce076d1533de: Lost carrier
Jun 1 18:10:19 docker systemd-networkd[180]: br-ce076d1533de: Gained IPv6LL

[iddqd-dev]

docker-compose up --force-recreate

[truelai]

Thanks. Turns out this was an issue with the container (LXD/LXC) and cgroup v2.

runc was throwing error due to inability to load eBPF

Moving install to KVM guest fixed the issue.

thanks, @iddqd-dev

[truelai]

solved

[iddqd-dev]

Thanks. Turns out this was an issue with the container (LXD/LXC) and cgroup v2.

runc was throwing error due to inability to load eBPF

Moving install to KVM guest fixed the issue.

thanks, @iddqd-dev

Np collegue