CRIU error writing to cpuset.cpus: Permission denied

[IronMatt97]

Description

Hello, I am new here and I'm trying to use Criu in order to checkpoint and restore a container from an host to another one. In particular I am using two identical virtual machines (the second cloned from the first): on the first vm I start a custom container, checkpoint it and send it (via http) to the second one. When I try to restore it (from the second vm) I get an error about a lack of permissions. I want to precise:

• I am running as root on both vms
• Both vms have the same container image
• I am using Podman's CRIU Api
• Both vms run Ubuntu 20.04.5 LTS

The basic checkpoint/restore procedure works well if I don't change host. So if from the first vm I checkpoint and then restore the container it works fine (same thing for the second one). The problem occurs only if I try to migrate the container.

CRIU logs and information:

CRIU full restore log:

(00.000012) Version: 3.17 (gitid v3.17-109-g1c6517a88)
(00.000024) Running on lubuntu2004 Linux 5.4.0-132-generic #148-Ubuntu SMP Mon Oct 17 16:02:06 UTC 2022 x86_64
(00.000026) Effective capability 40 missing
(00.000035) File /run/criu.kdat does not exist
(00.000042) sockets: Probing sock diag modules
(00.013716) sockets: Done probing
(00.020073) Pagemap is fully functional
(00.020093) Found anon-shmem device at 1
(00.020110) Hugetlb size 2 Mb is supported but cannot get dev's number
(00.020114) Reset 2282's dirty tracking
(00.020143)  ... done
(00.020156) Dirty track supported on kernel
(00.020191) Found task size of 7ffffffff000
(00.021339) Restoring netdev veth idx 10
(00.021461) Dumping netns links
(00.021478)     LD: Got link 1, type 772
(00.021480)     LD: Got link 10, type 1
(00.161387) No MOVE_MOUNT_SET_GROUP kernel feature
(00.178988) vdso: Parsing at 7ffe47f7d000 7ffe47f7e000
(00.178998) vdso: PT_LOAD p_vaddr: 0
(00.179000) vdso: DT_HASH: 120
(00.179001) vdso: DT_STRTAB: 2d8
(00.179003) vdso: DT_SYMTAB: 1b8
(00.179004) vdso: DT_STRSZ: 72
(00.179006) vdso: DT_SYMENT: 18
(00.179007) vdso: nbucket 3 nchain c bucket 7ffe47f7d128 chain 7ffe47f7d134
(00.179010) vdso: rt [vdso] 7ffe47f7d000-7ffe47f7e000 [vvar] 7ffe47f7a000-7ffe47f7d000
(00.179524) vdso: Parsing at 7f24e6aa6000 7f24e6aa8000
(00.179530) vdso: PT_LOAD p_vaddr: 0
(00.179533) vdso: DT_HASH: b4
(00.179534) vdso: DT_STRTAB: 1f0
(00.179536) vdso: DT_SYMTAB: 140
(00.179537) vdso: DT_STRSZ: c0
(00.179539) vdso: DT_SYMENT: 10
(00.179540) vdso: nbucket 3 nchain b bucket 7f24e6aa60bc chain 7f24e6aa60c8
(00.179543) vdso: compat [vdso] 7ffe47f80000-7ffe47f82000 [vvar] 7ffe47f7d000-7ffe47f80000
(00.179842) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id 11th Gen Intel(R) Core(TM) i7-11370H @ 3.30GHz
(00.179851) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 2440
(00.179855) cpu: fpu: x87 floating point registers     xstate_offsets      0 / 0      xstate_sizes    160 / 160   
(00.179857) cpu: fpu: AVX registers                    xstate_offsets    576 / 576    xstate_sizes    256 / 256   
(00.181409) Time namespaces are not supported.
(00.181472) No openat2 syscall support
(00.181674) ptrace(PTRACE_GET_RSEQ_CONFIGURATION) is not supported
(00.181805) Found mmap_min_addr 0x10000
(00.181815) files stat: fs/nr_open 1048576
(00.181891) Will dump/restore TCP connections
(00.181894) mnt-v2: Mounts-v2 requires MOVE_MOUNT_SET_GROUP support
(00.181896) Mount engine fallback to --mntns-compat-mode mode
(00.181905) rlimit: RLIMIT_NOFILE unlimited for self
(00.182300) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:0 xsavec:0 xgetbv1:0 xsaves:0
(00.182315) kernel pid_max=4194304
(00.182317) Reading image tree
(00.182332) Add mnt ns 13 pid 1
(00.182335) Add net ns 10 pid 1
(00.182337) Add pid ns 9 pid 1
(00.182339) pstree pid_max=1
(00.182343) Will restore in 6c020000 namespaces
(00.182344) NS mask to use 6c020000
(00.182363) Collecting 51/56 (flags 3)
(00.182369) No memfd.img image
(00.182371)  `- ... done
(00.182372) Collecting 40/54 (flags 2)
(00.182381) Collected [usr/local/bin/python3.10] ID 0x1
(00.182383) Collected [usr/local/lib/python3.10/lib-dynload/unicodedata.cpython-310-x86_64-linux-gnu.so] ID 0x2
(00.182385) Collected [lib/libcrypto.so.1.1] ID 0x3
(00.182388) Collected [usr/local/lib/python3.10/lib-dynload/_json.cpython-310-x86_64-linux-gnu.so] ID 0x4
(00.182390) Collected [usr/lib/liblzma.so.5.2.5] ID 0x5
(00.182391) Collected [usr/local/lib/python3.10/lib-dynload/_lzma.cpython-310-x86_64-linux-gnu.so] ID 0x6
(00.182393) Collected [usr/lib/libbz2.so.1.0.8] ID 0x7
(00.182395) Collected [usr/local/lib/python3.10/lib-dynload/_bz2.cpython-310-x86_64-linux-gnu.so] ID 0x8
(00.182397) Collected [usr/local/lib/python3.10/lib-dynload/zlib.cpython-310-x86_64-linux-gnu.so] ID 0x9
(00.182399) Collected [lib/libssl.so.1.1] ID 0xa
(00.182401) Collected [usr/local/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so] ID 0xb
(00.182403) Collected [lib/libz.so.1.2.12] ID 0xc
(00.182405) Collected [usr/local/lib/python3.10/lib-dynload/binascii.cpython-310-x86_64-linux-gnu.so] ID 0xd
(00.182414) Collected [usr/local/lib/python3.10/lib-dynload/_struct.cpython-310-x86_64-linux-gnu.so] ID 0xe
(00.182416) Collected [usr/local/lib/python3.10/lib-dynload/select.cpython-310-x86_64-linux-gnu.so] ID 0xf
(00.182418) Collected [usr/local/lib/python3.10/lib-dynload/_socket.cpython-310-x86_64-linux-gnu.so] ID 0x10
(00.182420) Collected [usr/local/lib/python3.10/lib-dynload/_sha512.cpython-310-x86_64-linux-gnu.so] ID 0x11
(00.182422) Collected [usr/local/lib/python3.10/lib-dynload/_random.cpython-310-x86_64-linux-gnu.so] ID 0x12
(00.182423) Collected [usr/local/lib/python3.10/lib-dynload/_bisect.cpython-310-x86_64-linux-gnu.so] ID 0x13
(00.182425) Collected [usr/local/lib/python3.10/lib-dynload/_datetime.cpython-310-x86_64-linux-gnu.so] ID 0x14
(00.182427) Collected [usr/local/lib/python3.10/lib-dynload/math.cpython-310-x86_64-linux-gnu.so] ID 0x15
(00.182430) Collected [usr/local/lib/python3.10/lib-dynload/array.cpython-310-x86_64-linux-gnu.so] ID 0x16
(00.182432) Collected [usr/local/lib/libpython3.10.so.1.0] ID 0x17
(00.182437) Collected [lib/ld-musl-x86_64.so.1] ID 0x18
(00.182439) Collected [dev/pts/0] ID 0x1a
(00.182451) Collected [.] ID 0x1d
(00.182455) Collected [.] ID 0x1e
(00.182457)  `- ... done
(00.182459) Collecting 46/68 (flags 0)
(00.182463) No remap-fpath.img image
(00.182464)  `- ... done
(00.182468) No apparmor.img image
(00.182506) cg: Preparing cgroups yard (cgroups restore mode 0x4)
(00.184240) cg: Opening .criu.cgyard.gWusXK as cg yard
(00.184250) cg:         Making controller dir .criu.cgyard.gWusXK/memory (memory)
(00.185784) cg: Created cgroup dir memory/machine.slice/libpod-4cf56915face5cfc223de5500d3922b9b23c518a7128d7f080898d7c3bc50afc.scope
(00.185854) cg: Restore special props
(00.185857) cg: Restoring cgroup property value [1] to [memory/machine.slice/libpod-4cf56915face5cfc223de5500d3922b9b23c518a7128d7f080898d7c3bc50afc.scope/memory.use_hierarchy]
(00.185864) cg: Restoring cgroup property value [9223372036854771712] to [memory/machine.slice/libpod-4cf56915face5cfc223de5500d3922b9b23c518a7128d7f080898d7c3bc50afc.scope/memory.kmem.limit_in_bytes]
(00.185874) cg:         Making controller dir .criu.cgyard.gWusXK/freezer (freezer)
(00.187462) cg: Created cgroup dir freezer/machine.slice/libpod-4cf56915face5cfc223de5500d3922b9b23c518a7128d7f080898d7c3bc50afc.scope
(00.187475) cg: Restore special props
(00.187485) cg:         Making controller dir .criu.cgyard.gWusXK/perf_event (perf_event)
(00.189011) cg: Created cgroup dir perf_event/machine.slice/libpod-4cf56915face5cfc223de5500d3922b9b23c518a7128d7f080898d7c3bc50afc.scope
(00.189018) cg: Restore special props
(00.189022) cg:         Making controller dir .criu.cgyard.gWusXK/cpuset (cpuset)
(00.202045) cg: Created cgroup dir cpuset/machine.slice/libpod-4cf56915face5cfc223de5500d3922b9b23c518a7128d7f080898d7c3bc50afc.scope
(00.202064) cg: Restore special props
(00.202068) cg: Restoring cgroup property value [0-1] to [cpuset/machine.slice/libpod-4cf56915face5cfc223de5500d3922b9b23c518a7128d7f080898d7c3bc50afc.scope/cpuset.cpus]
(00.202088) Error (criu/cgroup.c:1476): cg: Failed writing 0-1 to cpuset/machine.slice/libpod-4cf56915face5cfc223de5500d3922b9b23c518a7128d7f080898d7c3bc50afc.scope/cpuset.cpus: Permission denied
(00.202096) Error (criu/cgroup.c:1743): cg: Restoring cpuset.cpus special property failed
(00.202098) Error (criu/cgroup.c:1801): cg: Restoring special cpuset props failed!

Output of `criu --version`:

Version: 3.17
GitID: v3.17-109-g1c6517a88

Output of `criu check --all`:

Warn  (criu/cr-check.c:1244): clone3() with set_tid not supported
Error (criu/cr-check.c:1286): Time namespaces are not supported
Error (criu/cr-check.c:1296): IFLA_NEW_IFINDEX isn't supported
Warn  (criu/cr-check.c:1318): Pidfd store requires pidfd_getfd syscall which is not supported
Warn  (criu/cr-check.c:813): ptrace(PTRACE_GET_RSEQ_CONFIGURATION) isn't supported. C/R of processes which are using rseq() won't work.
Looks good but some kernel features are missing
which, depending on your process tree, may cause
dump or restore failure.

Thanks in advance

[adrianreber]

Congratulations for opening ticket 2000 👏 😉

A couple of questions:

• Do the VMs have the same number of CPUs?
• Can you try it on a system with cgroup v2?
• Which container runtime are you using? runc or crun?
  • If using crun can you add manage-cgroups ignore to /etc/criu/default.conf (not sure if that actually works)
  • If using runc can you add manage-cgroups ignore to /etc/criu/runc.conf
  • If you are using crun, maybe try runc with the option from above

[IronMatt97]

Thanks for the quick and precise answer, you really helped me out. 😊
I managed to fix the issue following what you suggested.
I will report all the steps I've done to fix my issue, hoping this will help future people coming here.

• Yes, both vms had 2 dedicated CPUs.
• I added to /etc/default/grub the string GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1", then run 'update-grub'. I don't know if this is enough to use cgroup v2 as you suggested, but it seems to have worked somehow with the other things I've done below.
• I was using crun as runtime, in particular the package crun_100:1.2-2_amd64

By the way, moving to runc helped.

In order to use runc instead of crun, I edited the file /usr/share/containers/containers.conf. In particular I uncommented some lines until the file looked like:

containers.conf file content:

# The containers configuration file specifies all of the available configuration
# command-line options/flags for container engine tools like Podman & Buildah,
# but in a TOML format that can be easily modified and versioned.

# Please refer to containers.conf(5) for details of all configuration options.
# Not all container engines implement all of the options.
# All of the options have hard coded defaults and these options will override
# the built in defaults. Users can then override these options via the command
# line. Container engines will read containers.conf files in up to three
# locations in the following order:
#  1. /usr/share/containers/containers.conf
#  2. /etc/containers/containers.conf
#  3. $HOME/.config/containers/containers.conf (Rootless containers ONLY)
#  Items specified in the latter containers.conf, if they exist, override the
# previous containers.conf settings, or the default settings.

[containers]

# List of annotation. Specified as
# "key = value"
# If it is empty or commented out, no annotations will be added
#
#annotations = []

# Used to change the name of the default AppArmor profile of container engine.
#
#apparmor_profile = "container-default"

# Default way to to create a cgroup namespace for the container
# Options are:
# `private` Create private Cgroup Namespace for the container.
# `host`    Share host Cgroup Namespace with the container.
#
#cgroupns = "private"

# Control container cgroup configuration
# Determines  whether  the  container will create CGroups.
# Options are:
# `enabled`   Enable cgroup support within container
# `disabled`  Disable cgroup support, will inherit cgroups from parent
# `no-conmon` Do not create a cgroup dedicated to conmon.
#
#cgroups = "enabled"

# List of default capabilities for containers. If it is empty or commented out,
# the default capabilities defined in the container engine will be added.
#
default_capabilities = [
  "CHOWN",
  "DAC_OVERRIDE",
  "FOWNER",
  "FSETID",
  "KILL",
  "NET_BIND_SERVICE",
  "SETFCAP",
  "SETGID",
  "SETPCAP",
  "SETUID",
  "SYS_CHROOT"
]

# A list of sysctls to be set in containers by default,
# specified as "name=value",
# for example:"net.ipv4.ping_group_range=0 0".
#
default_sysctls = [
  "net.ipv4.ping_group_range=0 0",
]

# A list of ulimits to be set in containers by default, specified as
# "<ulimit name>=<soft limit>:<hard limit>", for example:
# "nofile=1024:2048"
# See setrlimit(2) for a list of resource names.
# Any limit not specified here will be inherited from the process launching the
# container engine.
# Ulimits has limits for non privileged container engines.
#
#default_ulimits = [
#  "nofile=1280:2560",
#]

# List of devices. Specified as
# "<device-on-host>:<device-on-container>:<permissions>", for example:
# "/dev/sdc:/dev/xvdc:rwm".
# If it is empty or commented out, only the default devices will be used
#
#devices = []

# List of default DNS options to be added to /etc/resolv.conf inside of the container.
#
#dns_options = []

# List of default DNS search domains to be added to /etc/resolv.conf inside of the container.
#
#dns_searches = []

# Set default DNS servers.
# This option can be used to override the DNS configuration passed to the
# container. The special value "none" can be specified to disable creation of
# /etc/resolv.conf in the container.
# The /etc/resolv.conf file in the image will be used without changes.
#
#dns_servers = []

# Environment variable list for the conmon process; used for passing necessary
# environment variables to conmon or the runtime.
#
#env = [
#  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
#  "TERM=xterm",
#]

# Pass all host environment variables into the container.
#
#env_host = false

# Default proxy environment variables passed into the container.
# The environment variables passed in include:
# http_proxy, https_proxy, ftp_proxy, no_proxy, and the upper case versions of
# these. This option is needed when host system uses a proxy but container
# should not use proxy. Proxy environment variables specified for the container
# in any other way will override the values passed from the host.
#
#http_proxy = true

# Run an init inside the container that forwards signals and reaps processes.
#
#init = false

# Container init binary, if init=true, this is the init binary to be used for containers.
#
#init_path = "/usr/libexec/podman/catatonit"

# Default way to to create an IPC namespace (POSIX SysV IPC) for the container
# Options are:
# `private` Create private IPC Namespace for the container.
# `host`    Share host IPC Namespace with the container.
#
#ipcns = "private"

# keyring tells the container engine whether to create
# a kernel keyring for use within the container.
#
#keyring = true

# label tells the container engine whether to use container separation using
# MAC(SELinux) labeling or not.
# The label flag is ignored on label disabled systems.
#
#label = true

# Logging driver for the container. Available options: k8s-file and journald.
#
#log_driver = "k8s-file"

# Maximum size allowed for the container log file. Negative numbers indicate
# that no size limit is imposed. If positive, it must be >= 8192 to match or
# exceed conmon's read buffer. The file is truncated and re-opened so the
# limit is never exceeded.
#
log_size_max = -1

# Specifies default format tag for container log messages.
# This is useful for creating a specific tag for container log messages.
# Containers logs default to truncated container ID as a tag.
#
#log_tag = ""

# Default way to to create a Network namespace for the container
# Options are:
# `private` Create private Network Namespace for the container.
# `host`    Share host Network Namespace with the container.
# `none`    Containers do not use the network
#
#netns = "private"

# Create /etc/hosts for the container.  By default, container engine manage
# /etc/hosts, automatically adding  the container's  own  IP  address.
#
#no_hosts = false

# Default way to to create a PID namespace for the container
# Options are:
# `private` Create private PID Namespace for the container.
# `host`    Share host PID Namespace with the container.
#
#pidns = "private"

# Maximum number of processes allowed in a container.
#
#pids_limit = 2048

# Copy the content from the underlying image into the newly created volume
# when the container is created instead of when it is started. If false,
# the container engine will not copy the content until the container is started.
# Setting it to true may have negative performance implications.
#
#prepare_volume_on_create = false

# Indicates the networking to be used for rootless containers
#
#rootless_networking = "slirp4netns"

# Path to the seccomp.json profile which is used as the default seccomp profile
# for the runtime.
#
#seccomp_profile = "/usr/share/containers/seccomp.json"

# Size of /dev/shm. Specified as <number><unit>.
# Unit is optional, values:
# b (bytes), k (kilobytes), m (megabytes), or g (gigabytes).
# If the unit is omitted, the system uses bytes.
#
#shm_size = "65536k"

# Set timezone in container. Takes IANA timezones as well as "local",
# which sets the timezone in the container to match the host machine.
#
#tz = ""

# Set umask inside the container
#
#umask = "0022"

# Default way to to create a User namespace for the container
# Options are:
# `auto`        Create unique User Namespace for the container.
# `host`    Share host User Namespace with the container.
#
#userns = "host"

# Number of UIDs to allocate for the automatic container creation.
# UIDs are allocated from the "container" UIDs listed in
# /etc/subuid & /etc/subgid
#
#userns_size = 65536

# Default way to to create a UTS namespace for the container
# Options are:
# `private`        Create private UTS Namespace for the container.
# `host`    Share host UTS Namespace with the container.
#
#utsns = "private"

# List of volumes. Specified as
# "<directory-on-host>:<directory-in-container>:<options>", for example:
# "/db:/var/lib/db:ro".
# If it is empty or commented out, no volumes will be added
#
#volumes = []

# The network table contains settings pertaining to the management of
# CNI plugins.

[secrets]
#driver = "file"

[secrets.opts]
#root = "/example/directory"

[network]

# Path to directory where CNI plugin binaries are located.
#
cni_plugin_dirs = [
  "/usr/local/libexec/cni",
  "/usr/libexec/cni",
  "/usr/local/lib/cni",
  "/usr/lib/cni",
  "/opt/cni/bin",
]

# The network name of the default CNI network to attach pods to.
#
default_network = "podman"

# The default subnet for the default CNI network given in default_network.
# If a network with that name does not exist, a new network using that name and
# this subnet will be created.
# Must be a valid IPv4 CIDR prefix.
#
#default_subnet = "10.88.0.0/16"

# Path to the directory where CNI configuration files are located.
#
#network_config_dir = "/etc/cni/net.d/"

[engine]
# Index to the active service
#
#active_service = production

# Cgroup management implementation used for the runtime.
# Valid options "systemd" or "cgroupfs"
#
#cgroup_manager = "systemd"

# Environment variables to pass into conmon
#
#conmon_env_vars = [
#  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
#]

# Paths to look for the conmon container manager binary
#
#conmon_path = [
#  "/usr/libexec/podman/conmon",
#  "/usr/local/libexec/podman/conmon",
#  "/usr/local/lib/podman/conmon",
#  "/usr/bin/conmon",
#  "/usr/sbin/conmon",
#  "/usr/local/bin/conmon",
#  "/usr/local/sbin/conmon"
#]

# Specify the keys sequence used to detach a container.
# Format is a single character [a-Z] or a comma separated sequence of
# `ctrl-<value>`, where `<value>` is one of:
# `a-z`, `@`, `^`, `[`, `\`, `]`, `^` or `_`
#
#detach_keys = "ctrl-p,ctrl-q"

# Determines whether engine will reserve ports on the host when they are
# forwarded to containers. When enabled, when ports are forwarded to containers,
# ports are held open by as long as the container is running, ensuring that
# they cannot be reused by other programs on the host. However, this can cause
# significant memory usage if a container has many ports forwarded to it.
# Disabling this can save memory.
#
#enable_port_reservation = true

# Environment variables to be used when running the container engine (e.g., Podman, Buildah).
# For example "http_proxy=internal.proxy.company.com".
# Note these environment variables will not be used within the container.
# Set the env section under [containers] table, if you want to set environment variables for the container.
#
#env = []

# Selects which logging mechanism to use for container engine events.
# Valid values are `journald`, `file` and `none`.
#
#events_logger = "journald"

# A is a list of directories which are used to search for helper binaries.
#
#helper_binaries_dir = [
#  "/usr/local/libexec/podman",
#  "/usr/local/lib/podman",
#  "/usr/libexec/podman",
#  "/usr/lib/podman",
#]

# Path to OCI hooks directories for automatically executed hooks.
#
#hooks_dir = [
#  "/usr/share/containers/oci/hooks.d",
#]

# Manifest Type (oci, v2s2, or v2s1) to use when pulling, pushing, building
# container images. By default image pulled and pushed match the format of the
# source image. Building/committing defaults to OCI.
#
#image_default_format = ""

# Default transport method for pulling and pushing for images
#
#image_default_transport = "docker://"

# Maximum number of image layers to be copied (pulled/pushed) simultaneously.
# Not setting this field, or setting it to zero, will fall back to containers/image defaults.
#
#image_parallel_copies = 0

# Default command to run the infra container
#
#infra_command = "/pause"

# Infra (pause) container image name for pod infra containers.  When running a
# pod, we start a `pause` process in a container to hold open the namespaces
# associated with the  pod.  This container does nothing other then sleep,
# reserving the pods resources for the lifetime of the pod.
#
infra_image = "k8s.gcr.io/pause:3.4.1"

# Specify the locking mechanism to use; valid values are "shm" and "file".
# Change the default only if you are sure of what you are doing, in general
# "file" is useful only on platforms where cgo is not available for using the
# faster "shm" lock type.  You may need to run "podman system renumber" after
# you change the lock type.
#
lock_type = "shm"

# Indicates if Podman is running inside a VM via Podman Machine.
# Podman uses this value to do extra setup around networking from the
# container inside the VM to to host.
#
#machine_enabled = false

# MultiImageArchive - if true, the container engine allows for storing archives
# (e.g., of the docker-archive transport) with multiple images.  By default,
# Podman creates single-image archives.
#
#multi_image_archive = "false"

# Default engine namespace
# If engine is joined to a namespace, it will see only containers and pods
# that were created in the same namespace, and will create new containers and
# pods in that namespace.
# The default namespace is "", which corresponds to no namespace. When no
# namespace is set, all containers and pods are visible.
#
#namespace = ""

# Path to the slirp4netns binary
#
#network_cmd_path = ""

# Default options to pass to the slirp4netns binary.
# For example "allow_host_loopback=true"
#
#network_cmd_options = []

# Whether to use chroot instead of pivot_root in the runtime
#
#no_pivot_root = false

# Number of locks available for containers and pods.
# If this is changed, a lock renumber must be performed (e.g. with the
# 'podman system renumber' command).
#
num_locks = 2048

# Whether to pull new image before running a container
#
#pull_policy = "missing"

# Indicates whether the application should be running in remote mode. This flag modifies the
# --remote option on container engines. Setting the flag to true will default
# `podman --remote=true` for access to the remote Podman service.
#
#remote = false

# Default OCI runtime
#
runtime = "runc"

# List of the OCI runtimes that support --format=json.  When json is supported
# engine will use it for reporting nicer errors.
#
runtime_supports_json = ["crun", "runc", "kata", "runsc"]

# List of the OCI runtimes that supports running containers with KVM Separation.
#
#runtime_supports_kvm = ["kata"]

# List of the OCI runtimes that supports running containers without cgroups.
#
#runtime_supports_nocgroups = ["crun"]

# Directory for persistent engine files (database, etc)
# By default, this will be configured relative to where the containers/storage
# stores containers
# Uncomment to change location from this default
#
#static_dir = "/var/lib/containers/storage/libpod"

# Number of seconds to wait for container to exit before sending kill signal.
#
#stop_timeout = 10

# map of service destinations
#
#[service_destinations]
#  [service_destinations.production]
#     URI to access the Podman service
#     Examples:
#       rootless "unix://run/user/$UID/podman/podman.sock" (Default)
#       rootfull "unix://run/podman/podman.sock (Default)
#       remote rootless ssh://engineering.lab.company.com/run/user/1000/podman/podman.sock
#       remote rootfull ssh://root@10.10.1.136:22/run/podman/podman.sock
#
#    uri = "ssh://user@production.example.com/run/user/1001/podman/podman.sock"
#    Path to file containing ssh identity key
#    identity = "~/.ssh/id_rsa"

# Directory for temporary files. Must be tmpfs (wiped after reboot)
#
#tmp_dir = "/run/libpod"

# Directory for libpod named volumes.
# By default, this will be configured relative to where containers/storage
# stores containers.
# Uncomment to change location from this default.
#
#volume_path = "/var/lib/containers/storage/volumes"

# Paths to look for a valid OCI runtime (crun, runc, kata, runsc, etc)
[engine.runtimes]
#crun = [
#  "/usr/bin/crun",
#  "/usr/sbin/crun",
#  "/usr/local/bin/crun",
#  "/usr/local/sbin/crun",
#  "/sbin/crun",
#  "/bin/crun",
#  "/run/current-system/sw/bin/crun",
#]

#kata = [
#  "/usr/bin/kata-runtime",
#  "/usr/sbin/kata-runtime",
#  "/usr/local/bin/kata-runtime",
#  "/usr/local/sbin/kata-runtime",
#  "/sbin/kata-runtime",
#  "/bin/kata-runtime",
#  "/usr/bin/kata-qemu",
#  "/usr/bin/kata-fc",
#]

runc = [
  "/usr/bin/runc",
  "/usr/sbin/runc",
  "/usr/local/bin/runc",
  "/usr/local/sbin/runc",
  "/sbin/runc",
  "/bin/runc",
  "/usr/lib/cri-o-runc/sbin/runc",
]

#runsc = [
#  "/usr/bin/runsc",
#  "/usr/sbin/runsc",
#  "/usr/local/bin/runsc",
#  "/usr/local/sbin/runsc",
#  "/bin/runsc",
#  "/sbin/runsc",
#  "/run/current-system/sw/bin/runsc",
#]

[engine.volume_plugins]
#testplugin = "/run/podman/plugins/test.sock"

[machine]
# Number of CPU's a machine is created with.
#
#cpus=1

# The size of the disk in GB created when init-ing a podman-machine VM.
#
#disk_size=10

# The image used when creating a podman-machine VM.
#
#image = "testing"

# Memory in MB a machine is created with.
#
#memory=2048

# The [machine] table MUST be the last entry in this file.
# (Unless another table is added)
# TOML does not provide a way to end a table other than a further table being
# defined, so every key hereafter will be part of [machine] and not the
# main config.

I checked to be actually using runc running the command 'podman info | grep ociRuntime' and got:

podman ociRuntime info output:

ociRuntime: name: runc package: runc_1.1.0-0ubuntu1~20.04.2_amd64 path: /usr/sbin/runc version: |- runc version 1.1.0-0ubuntu1~20.04.2 spec: 1.0.2-dev go: go1.16.2 libseccomp: 2.5.1

At this point, trying to run all the migration process worked fine.
Thank you a lot for the help @adrianreber ! 💪

[adrianreber]

Good to hear that it works. Thanks for your detailed description. Hopefully it helps somebody else.

This probably also means we need to fix something in crun.