New Docker version fails in sys container

[rotoro-cloud]

Hello.

I rebuilt some images where docker is installed in the sysbox container.

Now it refuse to work:

Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
42c077c10790: Pull complete
62c70f376f6a: Pull complete
915cc9bd79c2: Pull complete
75a963e94de0: Pull complete
7b1fab684d70: Pull complete
db24d06d5af4: Pull complete
Digest: sha256:2bcabc23b45489fb0885d69a06ba1d648aeda973fae7bb981bafbb884165e514
Status: Downloaded newer image for nginx:latest
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/rdma/docker/c3c2f2af2d6af9430e4db5efddb5d7d8e8edb163458643196fbe68c31c0b9646: permission denied: unknown.
ERRO[0010] error waiting for container: context canceled

The docker version inside syscontainer:

Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:02 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:45:53 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

If I use my old environments all working fine.
This docker version is working.

 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:01:03 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.6
  GitCommit:        10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc:
  Version:          1.1.2
  GitCommit:        v1.1.2-0-ga916309
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

But if I update containerd.io, it stop working

My Dockerfile:

# Systemd installation

RUN [ -z "$(apt-get indextargets)" ]

RUN set -xe   && echo '#!/bin/sh' > /usr/sbin/policy-rc.d  && echo 'exit 101' >> /usr/sbin/policy-rc.d  && chmod +x /usr/sbin/policy-rc.d   && dpkg-divert --local --rename --add /sbin/initctl  && cp -a /usr/sbin/policy-rc.d /sbin/initctl  && sed -i 's/^exit.*/exit 0/' /sbin/initctl   && echo 'force-unsafe-io' > /etc/dpkg/dpkg.cfg.d/docker-apt-speedup   && echo 'DPkg::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };' > /etc/apt/apt.conf.d/docker-clean  && echo 'APT::Update::Post-Invoke { "rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true"; };' >> /etc/apt/apt.conf.d/docker-clean  && echo 'Dir::Cache::pkgcache ""; Dir::Cache::srcpkgcache "";' >> /etc/apt/apt.conf.d/docker-clean   && echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/docker-no-languages   && echo 'Acquire::GzipIndexes "true"; Acquire::CompressionTypes::Order:: "gz";' > /etc/apt/apt.conf.d/docker-gzip-indexes   && echo 'Apt::AutoRemove::SuggestsImportant "false";' > /etc/apt/apt.conf.d/docker-autoremove-suggests

RUN mkdir -p /run/systemd && echo 'docker' > /run/systemd/container

RUN echo "root:controlpass" | chpasswd

RUN apt-get update \
       && apt-get install -y --no-install-recommends  \
            systemd                                     \
            systemd-sysv                                \
            libsystemd0                                 \
            ca-certificates                             \
            dbus                                        \
            iptables                                    \
            iproute2                                    \
            kmod                                        \
            locales                                     \
            sudo                                        \
            udev                                        \
# Prevents journald from reading kernel messages from /dev/kmsg
       && echo "ReadKMsg=no" >> /etc/systemd/journald.conf \
# Housekeeping
       && apt-get clean -y \
       && rm -rf \
               /var/cache/debconf/* \
               /var/lib/apt/lists/* \
               /var/log/*           \
               /tmp/*               \
               /var/tmp/*           \
               /usr/share/doc/*     \
               /usr/share/man/*     \
               /usr/share/local/*   \
       && useradd --create-home --shell /bin/bash admin \
       && echo "admin:admin" | chpasswd \
       && adduser admin sudo

# Make use of stopsignal (instead of sigterm) to stop systemd containers.
STOPSIGNAL SIGRTMIN+3
# Set systemd as entrypoint.
ENTRYPOINT [ "/sbin/init", "--log-level=err" ]

# Set locale
RUN sed -i 's/# \(en_US\.UTF-8 .*\)/\1/' /etc/locale.gen && locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

RUN apt-get update && apt-get install --no-install-recommends -y                        \
       apt-transport-https                                                              \
       ca-certificates                                                                  \
       curl                                                                             \
       gnupg-agent                                                                      \
       software-properties-common                                                       \
    && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -          \
    && apt-key fingerprint 0EBFCD88                                                     \
    && add-apt-repository                                                               \
       "deb [arch=amd64] https://download.docker.com/linux/ubuntu                       \
       $(lsb_release -cs)                                                               \
       stable"                                                                          \
    && apt-get update                                                                   \
    && apt-get install --no-install-recommends -y docker-ce docker-ce-cli containerd.io \
    && rm -rf /var/lib/apt/lists/*                                                      \
    # Add user "admin" to the Docker group                                              \
    && usermod -a -G docker admin

[rodnymolina]

Hi @rotoro-cloud, I suspect you may be running a Sysbox version older than v0.5.2 and that you are reproducing issue #544. If that's the case, update Sysbox to the latest release and try again.

[rodnymolina]

@rotoro-cloud, please let us know if you're still facing this issue. I'll close it for now as a dup.

[lroehrs]

@rodnymolina We had the same issue with v0.5.0, with v0.5.2 is it fixed.