C/R Support for POSIX message queues

[tobse80]

Description

It seems that checkpoint/restore for POSIX message queues is (still) not supported. Checkpointing applications using such queues fail:

(00.149697) Error (criu/files-reg.c:1710): Can't lookup mount=413 for fd=3 path=/mq0
(00.149706) Error (criu/cr-dump.c:1635): Dump files (pid: 13273) failed with -1(00.186459) Error (criu/cr-dump.c:2053): Dumping FAILED.
(00.186459) Error (criu/cr-dump.c:2053): Dumping FAILED.

Steps to reproduce the issue:

1. Create test application mq0.py

   import posix_ipc as ipc
   from time import sleep
   
   mq = ipc.MessageQueue('/mq0', ipc.O_CREAT)
   
   while True:
      sleep(0.5)

   [Note that Python was chosen for writing a compact test program. One could use C instead.]

2. Prepare podman (or docker) container file with Python3 support and posix_ipc module:

   FROM python:3-slim-bookworm
   RUN apt-get update && apt-get install -y --no-install-recommends \
         python3 python3-pip python3-dev gcc \
      && rm -rf /var/lib/apt/lists/*
   
   RUN pip install --break-system-packages --no-cache-dir \
         posix-ipc
   
   COPY mq0.py /

3. Build test container

   sudo podman build -t test container

4. Run container

   sudo podman run -dt test python3 /mqtest.py

5. Checkpoint container

   sudo podman container checkpoint -l

Describe the results you received:

Checkpointing the container fails:

2023-10-13T14:04:48.324540Z: CRIU checkpointing failed -52.  Please check CRIU logfile /var/lib/containers/storage/overlay-containers/8b54dfcfd617302f2f48a8a16dbd75e4b9a539591ab3dd5c2d7625b0eeaf802f/userdata/dump.log
Error: `/usr/bin/crun checkpoint --image-path /var/lib/containers/storage/overlay-containers/8b54dfcfd617302f2f48a8a16dbd75e4b9a539591ab3dd5c2d7625b0eeaf802f/userdata/checkpoint --work-path /var/lib/containers/storage/overlay-containers/8b54dfcfd617302f2f48a8a16dbd75e4b9a539591ab3dd5c2d7625b0eeaf802f/userdata 8b54dfcfd617302f2f48a8a16dbd75e4b9a539591ab3dd5c2d7625b0eeaf802f` failed: exit status 1

Describe the results you expected:

Checkpoint (and restore) should also include POSIX message queues, as other IPC mechanisms like POSIX shared memories work without problems.

Additional tests:

When playing around with the test application by closing and/or unlinking the message queue after creation, criu's error messages vary a bit:

1. mq.close_fd() after creation:

   (00.016597) mnt:     419: 2f:/ @ ./dev/mqueue
   (00.016611) Error (criu/filesystems.c:649): mqueue isn't empty
   (00.090003) Error (criu/cr-dump.c:2053): Dumping FAILED.
   

2. mq.unlink() after creation:

   (00.105206) Error (criu/files-reg.c:1710): Can't lookup mount=413 for fd=3 path=/mq2 (deleted)
   (00.105215) Error (criu/cr-dump.c:1635): Dump files (pid: 14942) failed with -1
   (00.140750) Error (criu/cr-dump.c:2053): Dumping FAILED.
   

3. mq.close_fd() and mq.unlink() after creation:
   No error, checkpoint succeeds as expected (but w/o mq for the application).

CRIU logs and information:

CRIU full dump/restore logs:

(00.000000) Unable to get $HOME directory, local configuration file will not be used.
(00.000017) Version: 3.17.1 (gitid 0)
(00.000020) Running on debian Linux 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64
(00.000026) Loaded kdat cache from /run/criu.kdat
(00.000038) Hugetlb size 2 Mb is supported but cannot get dev's number
(00.000379) Hugetlb size 1024 Mb is supported but cannot get dev's number
(00.000640) ========================================
(00.000641) Dumping processes (pid: 13273)
(00.000642) ========================================
(00.000645) rlimit: RLIMIT_NOFILE unlimited for self
(00.000650) Running pre-dump scripts
(00.001021) irmap: Searching irmap cache in work dir
(00.001027) No irmap-cache image
(00.001029) irmap: Searching irmap cache in parent
(00.001031) No parent images directory provided
(00.001032) irmap: No irmap cache
(00.001040) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id 13th Gen Intel(R) Core(TM) i7-13800H
(00.001043) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 832
(00.001047) cpu: fpu: x87 floating point registers     xstate_offsets      0 / 0      xstate_sizes    160 / 160   
(00.001048) cpu: fpu: AVX registers                    xstate_offsets    576 / 576    xstate_sizes    256 / 256   
(00.001049) cpu: fpu:1 fxsr:1 xsave:1 xsaveopt:1 xsavec:1 xgetbv1:1 xsaves:1
(00.001108) cg-prop: Parsing controller "cpu"
(00.001110) cg-prop: 	Strategy "replace"
(00.001111) cg-prop: 	Property "cpu.shares"
(00.001111) cg-prop: 	Property "cpu.cfs_period_us"
(00.001112) cg-prop: 	Property "cpu.cfs_quota_us"
(00.001113) cg-prop: 	Property "cpu.rt_period_us"
(00.001113) cg-prop: 	Property "cpu.rt_runtime_us"
(00.001114) cg-prop: Parsing controller "memory"
(00.001115) cg-prop: 	Strategy "replace"
(00.001116) cg-prop: 	Property "memory.limit_in_bytes"
(00.001116) cg-prop: 	Property "memory.memsw.limit_in_bytes"
(00.001117) cg-prop: 	Property "memory.swappiness"
(00.001118) cg-prop: 	Property "memory.soft_limit_in_bytes"
(00.001118) cg-prop: 	Property "memory.move_charge_at_immigrate"
(00.001119) cg-prop: 	Property "memory.oom_control"
(00.001120) cg-prop: 	Property "memory.use_hierarchy"
(00.001121) cg-prop: 	Property "memory.kmem.limit_in_bytes"
(00.001121) cg-prop: 	Property "memory.kmem.tcp.limit_in_bytes"
(00.001122) cg-prop: Parsing controller "cpuset"
(00.001123) cg-prop: 	Strategy "replace"
(00.001123) cg-prop: 	Property "cpuset.cpus"
(00.001124) cg-prop: 	Property "cpuset.mems"
(00.001125) cg-prop: 	Property "cpuset.memory_migrate"
(00.001125) cg-prop: 	Property "cpuset.cpu_exclusive"
(00.001126) cg-prop: 	Property "cpuset.mem_exclusive"
(00.001127) cg-prop: 	Property "cpuset.mem_hardwall"
(00.001130) cg-prop: 	Property "cpuset.memory_spread_page"
(00.001131) cg-prop: 	Property "cpuset.memory_spread_slab"
(00.001131) cg-prop: 	Property "cpuset.sched_load_balance"
(00.001132) cg-prop: 	Property "cpuset.sched_relax_domain_level"
(00.001133) cg-prop: Parsing controller "blkio"
(00.001133) cg-prop: 	Strategy "replace"
(00.001134) cg-prop: 	Property "blkio.weight"
(00.001135) cg-prop: Parsing controller "freezer"
(00.001135) cg-prop: 	Strategy "replace"
(00.001136) cg-prop: Parsing controller "perf_event"
(00.001137) cg-prop: 	Strategy "replace"
(00.001138) cg-prop: Parsing controller "net_cls"
(00.001138) cg-prop: 	Strategy "replace"
(00.001139) cg-prop: 	Property "net_cls.classid"
(00.001140) cg-prop: Parsing controller "net_prio"
(00.001140) cg-prop: 	Strategy "replace"
(00.001141) cg-prop: 	Property "net_prio.ifpriomap"
(00.001142) cg-prop: Parsing controller "pids"
(00.001142) cg-prop: 	Strategy "replace"
(00.001143) cg-prop: 	Property "pids.max"
(00.001144) cg-prop: Parsing controller "devices"
(00.001144) cg-prop: 	Strategy "replace"
(00.001145) cg-prop: 	Property "devices.list"
(00.001154) Preparing image inventory (version 1)
(00.001166) Add pid ns 1 pid 13568
(00.001171) Add net ns 2 pid 13568
(00.001174) Add ipc ns 3 pid 13568
(00.001177) Add uts ns 4 pid 13568
(00.001180) Add time ns 5 pid 13568
(00.001186) Add mnt ns 6 pid 13568
(00.001191) Add user ns 7 pid 13568
(00.001194) Add cgroup ns 8 pid 13568
(00.001195) cg: Dumping cgroups for 13568
(00.001199) cg:  `- New css ID 1
(00.001199) cg:     `- [] -> [/user.slice/user-1000.slice/session-1.scope] [0]
(00.001200) cg: Set 1 is criu one
(00.001215) Detected cgroup V2 freezer
(00.001215) freezing processes: 100000 attempts with 100 ms steps
(00.001220) cgroup.freeze=0
(00.001240) cgroup.freeze=1
(00.101393) cgroup.freeze=1
(00.101442) freezing processes: 1 attempts done
(00.101468) SEIZE 13273: success
(00.101596) Seized task 13273, state 1
(00.101598) seccomp: Collected tid_real 13273 mode 0x2
(00.101607) Collected (0 attempts, 0 in_progress)
(00.101614) Collected (0 attempts, 0 in_progress)
(00.101617) Collected 13273 in 1 state
(00.101628) Will take pid namespace in the image
(00.101629) Add pid ns 9 pid 13273
(00.101632) Will take net namespace in the image
(00.101632) Add net ns 10 pid 13273
(00.101635) Will take ipc namespace in the image
(00.101636) Add ipc ns 11 pid 13273
(00.101639) Will take uts namespace in the image
(00.101640) Add uts ns 12 pid 13273
(00.101646) Will take mnt namespace in the image
(00.101647) Add mnt ns 13 pid 13273
(00.101651) Will take cgroup namespace in the image
(00.101652) Add cgroup ns 14 pid 13273
(00.101652) Lock network
(00.101653) Running network-lock scripts
(00.146950) 	type overlay source overlay mnt_id 414 s_dev 0x2d / @ ./ flags 0x200000 options lowerdir=/var/lib/containers/storage/overlay/l/FNQZPXF7PLEXAGNTXDBW6JQNGT:/var/lib/containers/storage/overlay/l/PNC3I3KINZJQJHVDHTCAYMMP6R:/var/lib/containers/storage/overlay/l/I2X7UMMQOETEGKRN4SMB5EZFBR:/var/lib/containers/storage/overlay/l/R32HH5B2ZQKTB2FXCWB4EIURFJ:/var/lib/containers/storage/overlay/l/DI4CV4TTOSDZKD4U4OZSI3RKZD:/var/lib/containers/storage/overlay/l/KA7P45AHL7TNBZ75BZZZG2NXAX:/var/lib/containers/storage/overlay/l/ZZWYQOTWRDG67TGGLQZMN5JI22:/var/lib/containers/storage/overlay/l/D47BHZMWCKAR475JGFZL7JXTLS,upperdir=/var/lib/containers/storage/overlay/edcb0bc810af416a92ef90d26a76a3e5fb9319c017a94b62373953eb91f0bc53/diff,workdir=/var/lib/containers/storage/overlay/edcb0bc810af416a92ef90d26a76a3e5fb9319c017a94b62373953eb91f0bc53/work
(00.146961) 	type proc source proc mnt_id 415 s_dev 0x30 / @ ./proc flags 0x20000e options 
(00.146963) 	type tmpfs source tmpfs mnt_id 416 s_dev 0x31 / @ ./dev flags 0x100000a options size=65536k,mode=755,inode64
(00.146974) 	type sysfs source sysfs mnt_id 417 s_dev 0x32 / @ ./sys flags 0x20000f options 
(00.146976) 	type devpts source devpts mnt_id 418 s_dev 0x33 / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=666
(00.146985) 	type mqueue source mqueue mnt_id 419 s_dev 0x2f / @ ./dev/mqueue flags 0x20000e options 
(00.146987) 	type tmpfs source tmpfs mnt_id 420 s_dev 0x17 /containers/storage/overlay-containers/9d82258b02044fa731f421961d86f2b2cf063fc67e246e6ebef3c8225cfccf66/userdata/resolv.conf @ ./etc/resolv.conf flags 0x200000 options size=400724k,nr_inodes=500901,mode=755,inode64
(00.146990) 	type tmpfs source tmpfs mnt_id 421 s_dev 0x17 /containers/storage/overlay-containers/9d82258b02044fa731f421961d86f2b2cf063fc67e246e6ebef3c8225cfccf66/userdata/hosts @ ./etc/hosts flags 0x200000 options size=400724k,nr_inodes=500901,mode=755,inode64
(00.146992) 	type tmpfs source shm mnt_id 422 s_dev 0x2c / @ ./dev/shm flags 0x20000e options size=64000k,inode64
(00.146994) 	type tmpfs source tmpfs mnt_id 423 s_dev 0x17 /containers/storage/overlay-containers/9d82258b02044fa731f421961d86f2b2cf063fc67e246e6ebef3c8225cfccf66/userdata/.containerenv @ ./run/.containerenv flags 0x200000 options size=400724k,nr_inodes=500901,mode=755,inode64
(00.147007) 	type tmpfs source tmpfs mnt_id 424 s_dev 0x17 /containers/storage/overlay-containers/9d82258b02044fa731f421961d86f2b2cf063fc67e246e6ebef3c8225cfccf66/userdata/hostname @ ./etc/hostname flags 0x200000 options size=400724k,nr_inodes=500901,mode=755,inode64
(00.147010) 	type ext4 source /dev/sda1 mnt_id 425 s_dev 0x800001 /var/lib/containers/storage/overlay-containers/9d82258b02044fa731f421961d86f2b2cf063fc67e246e6ebef3c8225cfccf66/userdata/notify @ ./run/notify flags 0x200000 options errors=remount-ro
(00.147015) 	type cgroup2 source cgroup2 mnt_id 426 s_dev 0x1a /machine.slice/libpod-9d82258b02044fa731f421961d86f2b2cf063fc67e246e6ebef3c8225cfccf66.scope/container @ ./sys/fs/cgroup flags 0x20000f options 
(00.147017) 	type tmpfs source tmpfs mnt_id 427 s_dev 0x34 / @ ./proc/acpi flags 0x200001 options size=0k,inode64
(00.147020) 	type devtmpfs source udev mnt_id 428 s_dev 0x5 /null @ ./proc/kcore flags 0x200001 options size=1980976k,nr_inodes=495244,mode=755,inode64
(00.147023) 	type devtmpfs source udev mnt_id 429 s_dev 0x5 /null @ ./proc/keys flags 0x200001 options size=1980976k,nr_inodes=495244,mode=755,inode64
(00.147025) 	type devtmpfs source udev mnt_id 430 s_dev 0x5 /null @ ./proc/timer_list flags 0x200001 options size=1980976k,nr_inodes=495244,mode=755,inode64
(00.147027) 	type tmpfs source tmpfs mnt_id 431 s_dev 0x35 / @ ./sys/firmware flags 0x200001 options size=0k,inode64
(00.147031) 	type tmpfs source tmpfs mnt_id 432 s_dev 0x36 / @ ./sys/dev/block flags 0x200001 options size=0k,inode64
(00.147033) 	type proc source proc mnt_id 433 s_dev 0x30 /bus @ ./proc/bus flags 0x200001 options 
(00.147035) 	type proc source proc mnt_id 434 s_dev 0x30 /fs @ ./proc/fs flags 0x200001 options 
(00.147038) 	type proc source proc mnt_id 435 s_dev 0x30 /irq @ ./proc/irq flags 0x200001 options 
(00.147040) 	type proc source proc mnt_id 436 s_dev 0x30 /sys @ ./proc/sys flags 0x200001 options 
(00.147041) 	type proc source proc mnt_id 437 s_dev 0x30 /sysrq-trigger @ ./proc/sysrq-trigger flags 0x200001 options 
(00.147044) 	type devpts source devpts mnt_id 382 s_dev 0x33 /0 @ ./dev/console flags 0x200000 options gid=5,mode=620,ptmxmode=666
(00.147046) mnt: Building mountpoints tree
(00.147047) mnt: 	Building plain mount tree
(00.147048) mnt: 		Working on 382->416
(00.147049) mnt: 		Working on 437->415
(00.147049) mnt: 		Working on 436->415
(00.147050) mnt: 		Working on 435->415
(00.147051) mnt: 		Working on 434->415
(00.147051) mnt: 		Working on 433->415
(00.147052) mnt: 		Working on 432->417
(00.147052) mnt: 		Working on 431->417
(00.147053) mnt: 		Working on 430->415
(00.147053) mnt: 		Working on 429->415
(00.147054) mnt: 		Working on 428->415
(00.147055) mnt: 		Working on 427->415
(00.147055) mnt: 		Working on 426->417
(00.147056) mnt: 		Working on 425->414
(00.147056) mnt: 		Working on 424->414
(00.147057) mnt: 		Working on 423->414
(00.147058) mnt: 		Working on 422->416
(00.147058) mnt: 		Working on 421->414
(00.147059) mnt: 		Working on 420->414
(00.147059) mnt: 		Working on 419->416
(00.147060) mnt: 		Working on 418->416
(00.147061) mnt: 		Working on 417->414
(00.147061) mnt: 		Working on 416->414
(00.147062) mnt: 		Working on 415->414
(00.147062) mnt: 		Working on 414->381
(00.147063) mnt: 	Resorting children of 414 in mount order
(00.147064) mnt: 	Resorting children of 425 in mount order
(00.147065) mnt: 	Resorting children of 424 in mount order
(00.147065) mnt: 	Resorting children of 423 in mount order
(00.147066) mnt: 	Resorting children of 421 in mount order
(00.147066) mnt: 	Resorting children of 420 in mount order
(00.147067) mnt: 	Resorting children of 417 in mount order
(00.147068) mnt: 	Resorting children of 432 in mount order
(00.147068) mnt: 	Resorting children of 426 in mount order
(00.147069) mnt: 	Resorting children of 431 in mount order
(00.147069) mnt: 	Resorting children of 416 in mount order
(00.147070) mnt: 	Resorting children of 382 in mount order
(00.147071) mnt: 	Resorting children of 422 in mount order
(00.147071) mnt: 	Resorting children of 419 in mount order
(00.147072) mnt: 	Resorting children of 418 in mount order
(00.147072) mnt: 	Resorting children of 415 in mount order
(00.147074) mnt: 	Resorting children of 437 in mount order
(00.147074) mnt: 	Resorting children of 436 in mount order
(00.147075) mnt: 	Resorting children of 435 in mount order
(00.147075) mnt: 	Resorting children of 434 in mount order
(00.147076) mnt: 	Resorting children of 433 in mount order
(00.147076) mnt: 	Resorting children of 430 in mount order
(00.147078) mnt: 	Resorting children of 429 in mount order
(00.147079) mnt: 	Resorting children of 428 in mount order
(00.147079) mnt: 	Resorting children of 427 in mount order
(00.147080) mnt: Done:
(00.147081) mnt: [./](414->381)
(00.147081) mnt:  [./run/notify](425->414)
(00.147082) mnt:  <--
(00.147083) mnt:  [./etc/hostname](424->414)
(00.147083) mnt:  <--
(00.147084) mnt:  [./run/.containerenv](423->414)
(00.147085) mnt:  <--
(00.147085) mnt:  [./etc/hosts](421->414)
(00.147086) mnt:  <--
(00.147086) mnt:  [./etc/resolv.conf](420->414)
(00.147087) mnt:  <--
(00.147088) mnt:  [./sys](417->414)
(00.147088) mnt:   [./sys/dev/block](432->417)
(00.147089) mnt:   <--
(00.147089) mnt:   [./sys/fs/cgroup](426->417)
(00.147090) mnt:   <--
(00.147091) mnt:   [./sys/firmware](431->417)
(00.147091) mnt:   <--
(00.147092) mnt:  <--
(00.147092) mnt:  [./dev](416->414)
(00.147093) mnt:   [./dev/console](382->416)
(00.147094) mnt:   <--
(00.147094) mnt:   [./dev/shm](422->416)
(00.147095) mnt:   <--
(00.147095) mnt:   [./dev/mqueue](419->416)
(00.147096) mnt:   <--
(00.147097) mnt:   [./dev/pts](418->416)
(00.147097) mnt:   <--
(00.147098) mnt:  <--
(00.147098) mnt:  [./proc](415->414)
(00.147099) mnt:   [./proc/sysrq-trigger](437->415)
(00.147100) mnt:   <--
(00.147100) mnt:   [./proc/sys](436->415)
(00.147101) mnt:   <--
(00.147102) mnt:   [./proc/irq](435->415)
(00.147102) mnt:   <--
(00.147103) mnt:   [./proc/fs](434->415)
(00.147103) mnt:   <--
(00.147104) mnt:   [./proc/bus](433->415)
(00.147105) mnt:   <--
(00.147105) mnt:   [./proc/timer_list](430->415)
(00.147106) mnt:   <--
(00.147106) mnt:   [./proc/keys](429->415)
(00.147107) mnt:   <--
(00.147108) mnt:   [./proc/kcore](428->415)
(00.147108) mnt:   <--
(00.147109) mnt:   [./proc/acpi](427->415)
(00.147109) mnt:   <--
(00.147110) mnt:  <--
(00.147111) mnt: <--
(00.147114) mnt: 	The mount 418 is bind for 382 (@./dev/pts -> @./dev/console)
(00.147115) mnt: 	The mount 436 is bind for 437 (@./proc/sys -> @./proc/sysrq-trigger)
(00.147116) mnt: 	The mount 435 is bind for 437 (@./proc/irq -> @./proc/sysrq-trigger)
(00.147116) mnt: 	The mount 434 is bind for 437 (@./proc/fs -> @./proc/sysrq-trigger)
(00.147117) mnt: 	The mount 433 is bind for 437 (@./proc/bus -> @./proc/sysrq-trigger)
(00.147118) mnt: 	The mount 415 is bind for 437 (@./proc -> @./proc/sysrq-trigger)
(00.147119) mnt: 	The mount 429 is bind for 430 (@./proc/keys -> @./proc/timer_list)
(00.147119) mnt: 	The mount 428 is bind for 430 (@./proc/kcore -> @./proc/timer_list)
(00.147120) mnt: 	The mount 423 is bind for 424 (@./run/.containerenv -> @./etc/hostname)
(00.147121) mnt: 	The mount 421 is bind for 424 (@./etc/hosts -> @./etc/hostname)
(00.147121) mnt: 	The mount 420 is bind for 424 (@./etc/resolv.conf -> @./etc/hostname)
(00.147124) mnt: Found /proc/timer_list mapping for ./proc/timer_list mountpoint
(00.147125) mnt: Found /proc/keys mapping for ./proc/keys mountpoint
(00.147126) mnt: Found /proc/kcore mapping for ./proc/kcore mountpoint
(00.147127) mnt: Found /run/notify mapping for ./run/notify mountpoint
(00.147127) mnt: Found /etc/hostname mapping for ./etc/hostname mountpoint
(00.147128) mnt: Found /run/.containerenv mapping for ./run/.containerenv mountpoint
(00.147129) mnt: Found /dev/shm mapping for ./dev/shm mountpoint
(00.147130) mnt: Found /etc/hosts mapping for ./etc/hosts mountpoint
(00.147130) mnt: Found /etc/resolv.conf mapping for ./etc/resolv.conf mountpoint
(00.147132) mnt: Inspecting sharing on 382 shared_id 0 master_id 0 (@./dev/console)
(00.147132) mnt: Inspecting sharing on 437 shared_id 0 master_id 0 (@./proc/sysrq-trigger)
(00.147133) mnt: Inspecting sharing on 436 shared_id 0 master_id 0 (@./proc/sys)
(00.147134) mnt: Inspecting sharing on 435 shared_id 0 master_id 0 (@./proc/irq)
(00.147134) mnt: Inspecting sharing on 434 shared_id 0 master_id 0 (@./proc/fs)
(00.147135) mnt: Inspecting sharing on 433 shared_id 0 master_id 0 (@./proc/bus)
(00.147136) mnt: Inspecting sharing on 432 shared_id 0 master_id 0 (@./sys/dev/block)
(00.147136) mnt: Inspecting sharing on 431 shared_id 0 master_id 0 (@./sys/firmware)
(00.147138) mnt: Inspecting sharing on 430 shared_id 0 master_id 0 (@./proc/timer_list)
(00.147139) mnt: Inspecting sharing on 429 shared_id 0 master_id 0 (@./proc/keys)
(00.147140) mnt: Inspecting sharing on 428 shared_id 0 master_id 0 (@./proc/kcore)
(00.147140) mnt: Inspecting sharing on 427 shared_id 0 master_id 0 (@./proc/acpi)
(00.147141) mnt: Inspecting sharing on 426 shared_id 0 master_id 0 (@./sys/fs/cgroup)
(00.147142) mnt: Inspecting sharing on 425 shared_id 0 master_id 0 (@./run/notify)
(00.147142) mnt: Inspecting sharing on 424 shared_id 0 master_id 0 (@./etc/hostname)
(00.147143) mnt: Inspecting sharing on 423 shared_id 0 master_id 0 (@./run/.containerenv)
(00.147143) mnt: Inspecting sharing on 422 shared_id 0 master_id 0 (@./dev/shm)
(00.147144) mnt: Inspecting sharing on 421 shared_id 0 master_id 0 (@./etc/hosts)
(00.147145) mnt: Inspecting sharing on 420 shared_id 0 master_id 0 (@./etc/resolv.conf)
(00.147145) mnt: Inspecting sharing on 419 shared_id 0 master_id 0 (@./dev/mqueue)
(00.147146) mnt: Inspecting sharing on 418 shared_id 0 master_id 0 (@./dev/pts)
(00.147146) mnt: Inspecting sharing on 417 shared_id 0 master_id 0 (@./sys)
(00.147147) mnt: Inspecting sharing on 416 shared_id 0 master_id 0 (@./dev)
(00.147152) mnt: Inspecting sharing on 415 shared_id 0 master_id 0 (@./proc)
(00.147153) mnt: Inspecting sharing on 414 shared_id 0 master_id 0 (@./)
(00.147156) Collecting netns 10/13273
(00.147157) The net[4026532137] netns is external
(00.147158) Switching to 13273's net for collecting sockets
(00.147196) unix: 	Collected: ino 60171 peer_ino 0 family    1 type    5 state  7 name null
(00.148152) netlink: Collect netlink sock 0xf94e
(00.148154) netlink: Collect netlink sock 0xeb0a
(00.148155) netlink: Collect netlink sock 0xf954
(00.148155) netlink: Collect netlink sock 0xf950
(00.148156) netlink: Collect netlink sock 0xf952
(00.148156) netlink: Collect netlink sock 0xf959
(00.148157) netlink: Collect netlink sock 0xf951
(00.148157) netlink: Collect netlink sock 0xf94f
(00.148160) Collecting pidns 9/13273
(00.148175) seccomp: Use SECCOMP_FILTER_FLAG_TSYNC for tid_real 13273
(00.148205) No parent images directory provided
(00.148217) 13273 has lsm profile containers-default-0.50.1
(00.148237) rmrf: removing .criu.temp-aa-policy.O7KG9b
(00.148256) ========================================
(00.148257) Dumping task (pid: 13273)
(00.148258) ========================================
(00.148259) Obtaining task stat ... 
(00.148270) 
(00.148270) Collecting mappings (pid: 13273)
(00.148271) ----------------------------------------
(00.148301) Found regular file mapping, OK
(00.148320) Dumping path for -3 fd via self 15 [/usr/local/bin/python3.12]
(00.148355) vma 562a7540f000 borrows vfi from previous 562a7540e000
(00.148359) vma 562a75410000 borrows vfi from previous 562a7540f000
(00.148361) vma 562a75411000 borrows vfi from previous 562a75410000
(00.148364) vma 562a75412000 borrows vfi from previous 562a75411000
(00.148379) Found regular file mapping, OK
(00.148385) Dumping path for -3 fd via self 15 [/usr/local/lib/python3.12/site-packages/posix_ipc.cpython-312-x86_64-linux-gnu.so]
(00.148402) vma 7f24c053d000 borrows vfi from previous 7f24c053a000
(00.148405) vma 7f24c0541000 borrows vfi from previous 7f24c053d000
(00.148408) vma 7f24c0543000 borrows vfi from previous 7f24c0541000
(00.148410) vma 7f24c0544000 borrows vfi from previous 7f24c0543000
(00.148438) Found regular file mapping, OK
(00.148442) Dumping path for -3 fd via self 15 [/usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache]
(00.148450) Only file size could be stored for validation for file /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
(00.148456) Found regular file mapping, OK
(00.148460) Dumping path for -3 fd via self 15 [/usr/lib/locale/C.utf8/LC_CTYPE]
(00.148464) Only file size could be stored for validation for file /usr/lib/locale/C.utf8/LC_CTYPE
(00.148472) Found regular file mapping, OK
(00.148476) Dumping path for -3 fd via self 15 [/usr/lib/x86_64-linux-gnu/libm.so.6]
(00.148487) vma 7f24c081b000 borrows vfi from previous 7f24c080b000
(00.148507) vma 7f24c088e000 borrows vfi from previous 7f24c081b000
(00.148509) vma 7f24c08e8000 borrows vfi from previous 7f24c088e000
(00.148512) vma 7f24c08e9000 borrows vfi from previous 7f24c08e8000
(00.148516) Found regular file mapping, OK
(00.148520) Dumping path for -3 fd via self 15 [/usr/lib/x86_64-linux-gnu/libc.so.6]
(00.148532) vma 7f24c0910000 borrows vfi from previous 7f24c08ea000
(00.148567) vma 7f24c0a65000 borrows vfi from previous 7f24c0910000
(00.148569) vma 7f24c0ab8000 borrows vfi from previous 7f24c0a65000
(00.148572) vma 7f24c0abc000 borrows vfi from previous 7f24c0ab8000
(00.148579) Found regular file mapping, OK
(00.148583) Dumping path for -3 fd via self 15 [/usr/local/lib/libpython3.12.so.1.0]
(00.148594) vma 7f24c0bca000 borrows vfi from previous 7f24c0acd000
(00.148623) vma 7f24c0db2000 borrows vfi from previous 7f24c0bca000
(00.148626) vma 7f24c0f04000 borrows vfi from previous 7f24c0db2000
(00.148629) vma 7f24c0f7b000 borrows vfi from previous 7f24c0f04000
(00.148637) Found regular file mapping, OK
(00.148641) Dumping path for -3 fd via self 15 [/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2]
(00.148658) vma 7f24c10ec000 borrows vfi from previous 7f24c10eb000
(00.148661) vma 7f24c1111000 borrows vfi from previous 7f24c10ec000
(00.148664) vma 7f24c111b000 borrows vfi from previous 7f24c1111000
(00.148667) vma 7f24c111d000 borrows vfi from previous 7f24c111b000
(00.148677) Collected, longest area occupies 614 pages
(00.148678) 0x562a7540e000-0x562a7540f000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp  shmid: 0x1
(00.148679) 0x562a7540f000-0x562a75410000 (4K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0x1000 reg fp  shmid: 0x1
(00.148680) 0x562a75410000-0x562a75411000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x2000 reg fp  shmid: 0x1
(00.148681) 0x562a75411000-0x562a75412000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x2000 reg fp  shmid: 0x1
(00.148681) 0x562a75412000-0x562a75413000 (4K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x3000 reg fp  shmid: 0x1
(00.148682) 0x562a75a78000-0x562a75b0b000 (588K) prot 0x3 flags 0x22 fdflags 0 st 0x221 off 0 reg heap ap  shmid: 0
(00.148683) 0x7f24c053a000-0x7f24c053d000 (12K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp  shmid: 0x2
(00.148684) 0x7f24c053d000-0x7f24c0541000 (16K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0x3000 reg fp  shmid: 0x2
(00.148685) 0x7f24c0541000-0x7f24c0543000 (8K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x7000 reg fp  shmid: 0x2
(00.148685) 0x7f24c0543000-0x7f24c0544000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x8000 reg fp  shmid: 0x2
(00.148686) 0x7f24c0544000-0x7f24c0545000 (4K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x9000 reg fp  shmid: 0x2
(00.148687) 0x7f24c0545000-0x7f24c07ab000 (2456K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap  shmid: 0
(00.148688) 0x7f24c07ab000-0x7f24c07b2000 (28K) prot 0x1 flags 0x1 fdflags 0 st 0x81 off 0 reg fs  shmid: 0x3
(00.148688) 0x7f24c07b2000-0x7f24c0809000 (348K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp  shmid: 0x4
(00.148689) 0x7f24c0809000-0x7f24c080b000 (8K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap  shmid: 0
(00.148690) 0x7f24c080b000-0x7f24c081b000 (64K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp  shmid: 0x5
(00.148691) 0x7f24c081b000-0x7f24c088e000 (460K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0x10000 reg fp  shmid: 0x5
(00.148691) 0x7f24c088e000-0x7f24c08e8000 (360K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x83000 reg fp  shmid: 0x5
(00.148692) 0x7f24c08e8000-0x7f24c08e9000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0xdc000 reg fp  shmid: 0x5
(00.148693) 0x7f24c08e9000-0x7f24c08ea000 (4K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0xdd000 reg fp  shmid: 0x5
(00.148693) 0x7f24c08ea000-0x7f24c0910000 (152K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp  shmid: 0x6
(00.148694) 0x7f24c0910000-0x7f24c0a65000 (1364K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0x26000 reg fp  shmid: 0x6
(00.148695) 0x7f24c0a65000-0x7f24c0ab8000 (332K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x17b000 reg fp  shmid: 0x6
(00.148697) 0x7f24c0ab8000-0x7f24c0abc000 (16K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x1ce000 reg fp  shmid: 0x6
(00.148698) 0x7f24c0abc000-0x7f24c0abe000 (8K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x1d2000 reg fp  shmid: 0x6
(00.148699) 0x7f24c0abe000-0x7f24c0acb000 (52K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap  shmid: 0
(00.148699) 0x7f24c0acd000-0x7f24c0bca000 (1012K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp  shmid: 0x7
(00.148700) 0x7f24c0bca000-0x7f24c0db2000 (1952K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0xfd000 reg fp  shmid: 0x7
(00.148701) 0x7f24c0db2000-0x7f24c0f04000 (1352K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x2e5000 reg fp  shmid: 0x7
(00.148702) 0x7f24c0f04000-0x7f24c0f7b000 (476K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x436000 reg fp  shmid: 0x7
(00.148702) 0x7f24c0f7b000-0x7f24c10e8000 (1460K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x4ad000 reg fp  shmid: 0x7
(00.148703) 0x7f24c10e8000-0x7f24c10eb000 (12K) prot 0x3 flags 0x22 fdflags 0 st 0x201 off 0 reg ap  shmid: 0
(00.148704) 0x7f24c10eb000-0x7f24c10ec000 (4K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0 reg fp  shmid: 0x8
(00.148704) 0x7f24c10ec000-0x7f24c1111000 (148K) prot 0x5 flags 0x2 fdflags 0 st 0x41 off 0x1000 reg fp  shmid: 0x8
(00.148705) 0x7f24c1111000-0x7f24c111b000 (40K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x26000 reg fp  shmid: 0x8
(00.148706) 0x7f24c111b000-0x7f24c111d000 (8K) prot 0x1 flags 0x2 fdflags 0 st 0x41 off 0x30000 reg fp  shmid: 0x8
(00.148707) 0x7f24c111d000-0x7f24c111f000 (8K) prot 0x3 flags 0x2 fdflags 0 st 0x41 off 0x32000 reg fp  shmid: 0x8
(00.148707) 0x7ffd93ffb000-0x7ffd9401c000 (132K) prot 0x3 flags 0x122 fdflags 0 st 0x201 off 0 reg ap  shmid: 0
(00.148708) 0x7ffd940e7000-0x7ffd940eb000 (16K) prot 0x1 flags 0x22 fdflags 0 st 0x1201 off 0 reg vvar ap  shmid: 0
(00.148709) 0x7ffd940eb000-0x7ffd940ed000 (8K) prot 0x5 flags 0x22 fdflags 0 st 0x209 off 0 reg vdso ap  shmid: 0
(00.148710) ----------------------------------------
(00.148710) 
(00.148711) Collecting fds (pid: 13273)
(00.148712) ----------------------------------------
(00.148719) Found 4 file descriptors
(00.148719) ----------------------------------------
(00.148724) Dump private signals of 13273
(00.148725) Dump shared signals of 13273
(00.148727) Dump rseq of 13273: ptr = 0x7f24c080a4a0 sign = 0x53053053
(00.148733) Parasite syscall_ip at 0x562a7540f000
(00.149148) Set up parasite blob using memfd
(00.149187) Putting parasite blob into 0x7f5ec7721000->0x7f24c0522000
(00.149303) Dumping general registers for 13273 in native mode
(00.149305) Dumping GP/FPU registers for 13273
(00.149311) x86: xsave runtime structure
(00.149312) x86: -----------------------
(00.149313) x86: cwd:0x37f swd:0 twd:0 fop:0 mxcsr:0x1fa0 mxcsr_mask:0xffff
(00.149314) x86: magic1:0x46505853 extended_size:836 xstate_bv:0x7 xstate_size:832
(00.149315) x86: xstate_bv: 0x7
(00.149316) x86: -----------------------
(00.149316) Putting tsock into pid 13273
(00.149377) Wait for parasite being daemonized...
(00.149378) Wait for ack 2 on daemon socket
pie: 1: Running daemon thread leader
pie: 1: __sent ack msg: 2 2 0
pie: 1: Daemon waits for command
(00.149444) Fetched ack: 2 2 0
(00.149445) Parasite 13273 has been switched to daemon mode
(00.149456) Sent msg to daemon 72 0 0
pie: 1: __fetched msg: 72 0 0
pie: 1: __sent ack msg: 72 72 0
pie: 1: Daemon waits for command
(00.149483) Wait for ack 72 on daemon socket
(00.149485) Fetched ack: 72 72 0
(00.149486) vdso: vDSO hint is reliable - omit checking
(00.149495) Sent msg to daemon 74 0 0
(00.149496) Wait for ack 74 on daemon socket
pie: 1: __fetched msg: 74 0 0
pie: 1: __sent ack msg: 74 74 0
pie: 1: Daemon waits for command
(00.149514) Fetched ack: 74 74 0
(00.149523) Sent msg to daemon 70 0 0
(00.149524) Wait for ack 70 on daemon socket
pie: 1: __fetched msg: 70 0 0
pie: 1: __sent ack msg: 70 70 0
pie: 1: Daemon waits for command
(00.149546) Fetched ack: 70 70 0
(00.149547) sid=1 pgid=1 pid=1
(00.149561) 
(00.149562) Dumping opened files (pid: 13273)
(00.149562) ----------------------------------------
(00.149573) Sent msg to daemon 71 0 0
pie: 1: __fetched msg: 71 0 0
pie: 1: __sent ack msg: 71 71 0
pie: 1: Daemon waits for command
(00.149593) Wait for ack 71 on daemon socket
(00.149594) Fetched ack: 71 71 0
(00.149605) 13273 fdinfo 0: pos:                0 flags:           100002/0
(00.149612) tty: Dumping tty 18 with id 0x9
(00.149614) Dumping path for 0 fd via self 18 [/dev/pts/0]
(00.149624) Sent msg to daemon 73 0 0
(00.149625) Wait for ack 73 on daemon socket
pie: 1: __fetched msg: 73 0 0
pie: 1: __sent ack msg: 73 73 0
pie: 1: Daemon waits for command
(00.149643) Fetched ack: 73 73 0
(00.149664) 13273 fdinfo 1: pos:                0 flags:           100002/0
(00.149672) 13273 fdinfo 2: pos:                0 flags:           100002/0
(00.149678) 13273 fdinfo 3: pos:                0 flags:                2/0x1
(00.149697) Error (criu/files-reg.c:1710): Can't lookup mount=413 for fd=3 path=/mq0
(00.149700) ----------------------------------------
(00.149706) Error (criu/cr-dump.c:1635): Dump files (pid: 13273) failed with -1
(00.149716) Waiting for 13273 to trap
(00.149731) Daemon 13273 exited trapping
(00.149733) Sent msg to daemon 3 0 0
(00.149736) Force no-breakpoints restore
(00.149758) 13273 was trapped
(00.149760) 13273 (native) is going to execute the syscall 45, required is 15
(00.149783) 13273 was trapped
(00.149784) `- Expecting exit
(00.149806) 13273 was trapped
(00.149807) 13273 (native) is going to execute the syscall 186, required is 15
(00.149829) 13273 was trapped
(00.149830) `- Expecting exit
(00.149853) 13273 was trapped
(00.149855) 13273 (native) is going to execute the syscall 1, required is 15
pie: 1: __fetched msg: 3 0 0
(00.149877) 13273 was trapped
(00.149879) `- Expecting exit
(00.149903) 13273 was trapped
(00.149905) 13273 (native) is going to execute the syscall 186, required is 15
(00.149929) 13273 was trapped
(00.149930) `- Expecting exit
(00.149952) 13273 was trapped
(00.149954) 13273 (native) is going to execute the syscall 186, required is 15
(00.149975) 13273 was trapped
(00.149976) `- Expecting exit
(00.149998) 13273 was trapped
(00.150000) 13273 (native) is going to execute the syscall 1, required is 15
pie: 1: 1: new_sp=0x7f24c0533ec8 ip 0x7f24c09b9341
(00.150022) 13273 was trapped
(00.150023) `- Expecting exit
(00.150044) 13273 was trapped
(00.150046) 13273 (native) is going to execute the syscall 3, required is 15
(00.150070) 13273 was trapped
(00.150071) `- Expecting exit
(00.150093) 13273 was trapped
(00.150094) 13273 (native) is going to execute the syscall 3, required is 15
(00.150116) 13273 was trapped
(00.150117) `- Expecting exit
(00.150139) 13273 was trapped
(00.150141) 13273 (native) is going to execute the syscall 15, required is 15
(00.150164) 13273 was stopped
(00.150276) Unlock network
(00.150282) Running network-unlock scripts
(00.186424) Unfreezing tasks into 1
(00.186439) 	Unseizing 13273 into 1
(00.186459) Error (criu/cr-dump.c:2053): Dumping FAILED.

Output of `criu --version`:

Version: 3.17.1

Output of `criu check --all`:

Looks good.

Additional environment details:

uname -a
Linux debian 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[tobse80]

Hi all, any news/comments/ideas on this?
Thanks!

[adrianreber]

Hi all, any news/comments/ideas on this?

No, I think mqueue checkpoint/restore unfortunately only works with empty mqueues at this point. Maybe this is something you could implement?

[tobse80]

No, I think mqueue checkpoint/restore unfortunately only works with empty mqueues at this point.

I somehow suspected that, but thanks for clarifying that.

Maybe this is something you could implement?

Well, I am quite new to this topic and have first to understand how C/R for the other filesystems work. Also, whether skipping this feature 7 years ago was due to lack of importance, or more to technical issues hard to solve. A quick research in the repo's git history didn't reveal any hints on that.

[Snorch]

skipping this feature 7 years ago was due to lack of importance, or more to technical issues hard to solve

I see one technical issue: we don't have ability to get messages from such queues without actually removing messages from queue, we need a kernel interface like we have for sockets receive with MSG_PEAK to be non-intrusive.

Still we can:

1) go over all IPC namespaces
  2) save content of /proc/sys/fs/mqueue/* files to images
  3) mount /dev/mqueue for this IPC namespace into some temporary directory on host
  4) go over files in this mount
    5) receive all messages with their priorities from file and save them in the same order to images
    6) put all messages back with same priorities and same order (in case we would need to release task after dump)
  7) umount temporary /dev/mqueue for this IPC namespace

In case criu segfaults, or is killed, or just has bad error handling between 5 and 6 - original container will have broken mqueue communication.

We can restore in similar way as the dump with inverted operations.

note: We will need to save/restore mq_getattr/mq_setattr for each mqueue.
note: Supporting mq_notify and mq_timedreceive may add some more complication.

And that seems it.

[snprajwal]

It looks like the problem and solution are both pretty well-defined, do you guys think this could be a potential project for GSoC 2024?

[kishore-ganesh]

I was starting to poke around this. Not sure if it is some kernel bug, but inside the container:
cat /proc/pid/fdinfo/3 (3 is the fd for the mqueue fd) has a mnt_id that doesn't exist in
cat /proc/pid/mountinfo

It seems a similar issue has been encountered by others: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1507463

 * This is here only to support the Linux Kernel between versions
 * 3.18 and 4.2. After that, this workaround is not needed anymore,
 * but it will work properly on both a kernel with and without the bug.
 *
 * When a process has a file open in an OverlayFS directory,
 * the information in /proc/<pid>/fd/<fd> and /proc/<pid>/fdinfo/<fd>
 * is wrong. We can't even rely on stat()-ing /proc/<pid>/fd/<fd> since
 * this will show us the wrong filesystem type.

I do see some comments in the code (criu/files.c) about wrong info for overlayfs for certain kernel versions, but my kernel is 5.19 (whereas the comment above suggests that such a bug last appeared in 4.2)

Also might be relevant: https://patchwork.kernel.org/project/linux-fsdevel/patch/20191121070613.4286-1-hu1.chen@intel.com/
Another instance of this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1880842
https://patchwork.ozlabs.org/project/ubuntu-kernel/patch/20210426081121.37363-1-alexander@mihalicyn.com/#2673002
UPDATE: I was looking through the code in __lookup_overlayfs..seems it uses tries to find the right mount id by searching mountpoints using device number. However, this breaks in this case since even that is different for some reason?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.