CRIU dump failed in docker:Error (criu/util.c:626): execvp("iptables-restore", ...) failed: No such file or directory

[chenguanxi12138]

When I use criu to save the container, I get the following error logs:

(00.000037) Version: 3.17 (gitid v3.17)
(00.000060) Running on VSWE2 Linux 5.4.91-rt50-CGELV7.04.10B3-BSP1.0.0 #2 SMP PREEMPT Thu Jan 25 19:24:53 CST 2024 aarch64
(00.000070) File /run/criu.kdat does not exist
(00.000281) sockets: Probing sock diag modules
(00.000696) Error (criu/sockets.c:209): sockets: Diag module missing (-2)
(00.000826) Error (criu/sockets.c:209): sockets: Diag module missing (-2)
(00.000963) Error (criu/sockets.c:209): sockets: Diag module missing (-2)
(00.001257) sockets: Done probing
(00.001801) Error (criu/util.c:641): exited, status=1
(00.002620) Error (criu/util.c:641): exited, status=1
(00.002668) Pagemap is fully functional
(00.002703) Found anon-shmem device at 1
(00.002774) Hugetlb size 0 Mb is supported but cannot get dev's number
(00.002802) Found hugetlb device at d
(00.002814) Hugetlb size 32 Mb is supported but cannot get dev's number
(00.002826) Hugetlb size 1024 Mb is supported but cannot get dev's number
(00.002831) Reset 10525's dirty tracking
(00.002855) ... done
(00.002875) Dirty tracking support is OFF
(00.002936) Found task size of 1000000000000
(00.022482) Restoring netdev veth idx 10
(00.022994) Dumping netns links
(00.023059) LD: Got link 1, type 772
(00.023064) LD: Got link 2, type 776
(00.023067) LD: Got link 10, type 1
(00.023579) uffd: Lazy pages are not available: Function not implemented
(00.023842) vdso: Parsing at ffff8d4da000 ffff8d4db000
(00.023846) vdso: PT_LOAD p_vaddr: 0
(00.023849) vdso: DT_HASH: 120
(00.023851) vdso: DT_STRTAB: 1e0
(00.023853) vdso: DT_SYMTAB: 150
(00.023855) vdso: DT_STRSZ: 77
(00.023858) vdso: DT_SYMENT: 18
(00.023860) vdso: nbucket 3 nchain 6 bucket ffff8d4da128 chain ffff8d4da134
(00.023863) vdso: rt [vdso] ffff8d4da000-ffff8d4db000 [vvar] ffff8d4d9000-ffff8d4da000
(00.024337) Time namespaces are not supported.
(00.024426) Warn (criu/kerndat.c:1453): CRIU was built without libnftables support
(00.024607) No MOVE_MOUNT_SET_GROUP kernel feature
(00.024748) No openat2 syscall support
(00.024960) ptrace(PTRACE_GET_RSEQ_CONFIGURATION) is not supported
(00.025212) Adjust mmap_min_addr 0x1000 -> 0x10000
(00.025216) Found mmap_min_addr 0x10000
(00.025299) files stat: fs/nr_open 1048576
(00.025393) Will dump/restore TCP connections
(00.025407) ========================================
(00.025411) Dumping processes (pid: 2411)
(00.025413) ========================================
(00.025419) rlimit: RLIMIT_NOFILE unlimited for self
(00.025423) Running pre-dump scripts
(00.025425) RPC
(00.025529) irmap: Searching irmap cache in work dir
(00.025543) No irmap-cache image
(00.025546) irmap: Searching irmap cache in parent
(00.025554) No parent images directory provided
(00.025556) irmap: No irmap cache
(00.025617) cg-prop: Parsing controller "cpu"
(00.025621) cg-prop: Strategy "replace"
(00.025625) cg-prop: Property "cpu.shares"
(00.025628) cg-prop: Property "cpu.cfs_period_us"
(00.025630) cg-prop: Property "cpu.cfs_quota_us"
(00.025632) cg-prop: Property "cpu.rt_period_us"
(00.025635) cg-prop: Property "cpu.rt_runtime_us"
(00.025637) cg-prop: Parsing controller "memory"
(00.025639) cg-prop: Strategy "replace"
(00.025642) cg-prop: Property "memory.limit_in_bytes"
(00.025644) cg-prop: Property "memory.memsw.limit_in_bytes"
(00.025646) cg-prop: Property "memory.swappiness"
(00.025648) cg-prop: Property "memory.soft_limit_in_bytes"
(00.025650) cg-prop: Property "memory.move_charge_at_immigrate"
(00.025653) cg-prop: Property "memory.oom_control"
(00.025655) cg-prop: Property "memory.use_hierarchy"
(00.025657) cg-prop: Property "memory.kmem.limit_in_bytes"
(00.025660) cg-prop: Property "memory.kmem.tcp.limit_in_bytes"
(00.025662) cg-prop: Parsing controller "cpuset"
(00.025664) cg-prop: Strategy "replace"
(00.025666) cg-prop: Property "cpuset.cpus"
(00.025668) cg-prop: Property "cpuset.mems"
(00.025671) cg-prop: Property "cpuset.memory_migrate"
(00.025673) cg-prop: Property "cpuset.cpu_exclusive"
(00.025675) cg-prop: Property "cpuset.mem_exclusive"
(00.025677) cg-prop: Property "cpuset.mem_hardwall"
(00.025679) cg-prop: Property "cpuset.memory_spread_page"
(00.025690) cg-prop: Property "cpuset.memory_spread_slab"
(00.025693) cg-prop: Property "cpuset.sched_load_balance"
(00.025695) cg-prop: Property "cpuset.sched_relax_domain_level"
(00.025697) cg-prop: Parsing controller "blkio"
(00.025700) cg-prop: Strategy "replace"
(00.025702) cg-prop: Property "blkio.weight"
(00.025704) cg-prop: Parsing controller "freezer"
(00.025706) cg-prop: Strategy "replace"
(00.025709) cg-prop: Parsing controller "perf_event"
(00.025711) cg-prop: Strategy "replace"
(00.025713) cg-prop: Parsing controller "net_cls"
(00.025715) cg-prop: Strategy "replace"
(00.025717) cg-prop: Property "net_cls.classid"
(00.025720) cg-prop: Parsing controller "net_prio"
(00.025722) cg-prop: Strategy "replace"
(00.025724) cg-prop: Property "net_prio.ifpriomap"
(00.025726) cg-prop: Parsing controller "pids"
(00.025728) cg-prop: Strategy "replace"
(00.025731) cg-prop: Property "pids.max"
(00.025733) cg-prop: Parsing controller "devices"
(00.025735) cg-prop: Strategy "replace"
(00.025737) cg-prop: Property "devices.list"
(00.025762) Preparing image inventory (version 1)
(00.025791) Add pid ns 1 pid 10525
(00.025802) Add net ns 2 pid 10525
(00.025813) Add ipc ns 3 pid 10525
(00.025823) Add uts ns 4 pid 10525
(00.025829) Add time ns 5 pid 10525
(00.025841) Add mnt ns 6 pid 10525
(00.025852) Add user ns 7 pid 10525
(00.025862) Add cgroup ns 8 pid 10525
(00.025864) cg: Dumping cgroups for 10525
(00.025875) cg: - New css ID 1 (00.025878) cg: - [cpu,cpuacct,blkio,memory,devices,freezer,pids] -> [/] [0]
(00.025880) cg: Set 1 is criu one
(00.025888) Detected cgroup V1 freezer
(00.025982) Seized task 2411, state 1
(00.025986) seccomp: Collected tid_real 2411 mode 0
(00.026008) Collected (4 attempts, 0 in_progress)
(00.026031) Seized task 2425, state 0
(00.026085) seccomp: Collected tid_real 2425 mode 0
(00.026099) Collected (4 attempts, 0 in_progress)
(00.026115) Collected (4 attempts, 0 in_progress)
(00.026120) Collected 2425 in 1 state
(00.026138) Collected (3 attempts, 0 in_progress)
(00.026143) Collected 2411 in 1 state
(00.026330) Will take pid namespace in the image
(00.026335) Add pid ns 9 pid 2411
(00.026346) Will take net namespace in the image
(00.026349) Add net ns 10 pid 2411
(00.026376) Will take ipc namespace in the image
(00.026379) Add ipc ns 11 pid 2411
(00.026388) Will take uts namespace in the image
(00.026390) Add uts ns 12 pid 2411
(00.026403) Will take mnt namespace in the image
(00.026405) Add mnt ns 13 pid 2411
(00.026479) Lock network
(00.026482) Running network-lock scripts
(00.026485) RPC
Error (criu/util.c:626): execvp("iptables-restore", ...) failed: No such file or directory
(00.027127) Error (criu/util.c:641): exited, status=1
Error (criu/util.c:626): execvp("ip6tables-restore", ...) failed: No such file or directory
(00.027741) Error (criu/util.c:641): exited, status=1
(00.027757) Error (criu/net.c:3071): Locking network failed: iptables-restore returned -1. This may be connected to disabled CONFIG_NETFILTER_XT_MARK kernel build config option.
(00.027774) Unlock network
(00.027777) Running network-unlock scripts
(00.027779) RPC
Error (criu/util.c:626): execvp("iptables-restore", ...) failed: No such file or directory
(00.028444) Error (criu/util.c:641): exited, status=1
Error (criu/util.c:626): execvp("ip6tables-restore", ...) failed: No such file or directory
(00.029038) Error (criu/util.c:641): exited, status=1
(00.029058) Unfreezing tasks into 1
(00.029061) Unseizing 2411 into 1
(00.029068) Unseizing 2425 into 1
(00.029091) Error (criu/cr-dump.c:2053): Dumping FAILED.

How can I solve this problem，Please take a look at it for me. Thank you

[rst0git]

(00.028444) Error (criu/util.c:641): exited, status=1
Error (criu/util.c:626): execvp("ip6tables-restore", ...) failed: No such file or directory
(00.029038) Error (criu/util.c:641): exited, status=1

@chenguanxi12138 It looks like iptables is not installed. Could you try to install it?

[chenguanxi12138]

(00.028444) Error (criu/util.c:641): exited, status=1
Error (criu/util.c:626): execvp("ip6tables-restore", ...) failed: No such file or directory
(00.029038) Error (criu/util.c:641): exited, status=1

@chenguanxi12138 It looks like iptables is not installed. Could you try to install it?

I tried to install iptables and turned on the relevant kernel configurations (I'm not sure if they are correct) but I get the following error logs：
(00.000058) Running on VSWE2 Linux 5.4.91-rt50-CGELV7.04.10B3-BSP1.0.0 #8 SMP PREEMPT Sun Jan 28 16:17:19 CST 2024 aarch64
(00.000069) File /run/criu.kdat does not exist
(00.000269) sockets: Probing sock diag modules
(00.000659) Error (criu/sockets.c:209): sockets: Diag module missing (-2)
(00.000790) Error (criu/sockets.c:209): sockets: Diag module missing (-2)
(00.000918) Error (criu/sockets.c:209): sockets: Diag module missing (-2)
(00.001190) sockets: Done probing
(00.003757) Pagemap is fully functional
(00.003797) Found anon-shmem device at 1
(00.003859) Hugetlb size 0 Mb is supported but cannot get dev's number
(00.003885) Found hugetlb device at d
(00.003897) Hugetlb size 32 Mb is supported but cannot get dev's number
(00.003909) Hugetlb size 1024 Mb is supported but cannot get dev's number
(00.003914) Reset 18684's dirty tracking
(00.003937) ... done
......
......
(00.153546) mnt: Inspecting sharing on 6352 shared_id 0 master_id 0 (@./sys/fs/cgroup)
(00.153548) mnt: Inspecting sharing on 6351 shared_id 0 master_id 0 (@./sys)
(00.153550) mnt: Inspecting sharing on 6343 shared_id 0 master_id 0 (@./dev/pts)
(00.153552) mnt: Inspecting sharing on 6341 shared_id 0 master_id 0 (@./dev)
(00.153554) mnt: Inspecting sharing on 6340 shared_id 0 master_id 0 (@./proc)
(00.153556) mnt: Inspecting sharing on 6337 shared_id 0 master_id 0 (@./)
(00.153562) Collecting netns 10/12171
(00.153565) Switching to 12171's net for collecting sockets
(00.153780) sockets: Sockects collect procedure family AF_UNIX proto IPPROTO_IP: -2
(00.158737) sockets: Sockects collect procedure family AF_INET proto IPPROTO_UDP: -2
(00.158887) sockets: Sockects collect procedure family AF_INET proto IPPROTO_UDPLITE: -2
(00.159019) sockets: Sockects collect procedure family AF_INET proto IPPROTO_RAW: -2
(00.163241) sockets: Sockects collect procedure family AF_INET6 proto IPPROTO_UDP: -2
(00.163368) sockets: Sockects collect procedure family AF_INET6 proto IPPROTO_UDPLITE: -2
(00.163491) sockets: Sockects collect procedure family AF_INET6 proto IPPROTO_RAW: -2
(00.163616) sockets: Sockects collect procedure family AF_PACKET proto IPPROTO_IP: -2
(00.163742) sockets: Sockects collect procedure family AF_NETLINK proto IPPROTO_RAW: -2
(00.163763) Unlock network
(00.163766) Running network-unlock scripts
(00.163769) �[38;17HRPC
(00.251897) Unfreezing tasks into 1
(00.251913) �[38;17HUnseizing 12171 into 1
(00.251923) �[38;17HUnseizing 12183 into 1
(00.251946) Error (criu/cr-dump.c:2053): Dumping FAILED.

Can you help me see what's wrong?I would also like to know if docker can use the checkpoint feature without relying on iptables, thanks!

[adrianreber]

One guess would be you are missing CONFIG_INET_*_DIAG options in your kernel.

[adrianreber]

I would also like to know if docker can use the checkpoint feature without relying on iptables, thanks!

Looking at the man page I see:

       --network-lock [mode]
           Set the method to be used for network locking/unlocking. Locking is done to ensure that tcp packets are dropped between dump and restore. This is done to avoid the kernel sending RST when a packet arrives destined for
           the dumped process.

           The mode may be one of the following:

           iptables
               Use iptables rules to drop the packets. This is the default if mode is not specified.

           nftables
               Use nftables rules to drop the packets.

           skip
               Don’t lock the network. If --tcp-close is not used, the network must be locked externally to allow CRIU to dump TCP connections.

skip could work.

[chenguanxi12138]

One guess would be you are missing CONFIG_INET_*_DIAG options in your kernel.

it works ! Now the checkpoint programme can go further.But I have a new problem.

(00.139151) Error (criu/util.c:641): exited, status=1
(00.139164) Error (criu/filesystems.c:426): Can't dump tmpfs content
(00.139239) Unlock network
(00.139242) Running network-unlock scripts
(00.139245) RPC
(00.192473) Unfreezing tasks into 1
(00.192495) Unseizing 29231 into 1
(00.192505) Unseizing 29249 into 1
(00.192527) Error (criu/cr-dump.c:2053): Dumping FAILED.

Please tell me what's going on. Thank you.

[adrianreber]

You are not including the relevant part of the error. I think you are missing a tar binary on your system.

[chenguanxi12138]

You are not including the relevant part of the error. I think you are missing a tar binary on your system.

I think this is the cause of the problem ---"tar: unrecognized option '--no-unquote' "

tar: unrecognized option '--no-unquote'
BusyBox v1.35.0(ZTEV6.01.10.50) multi-call binary.

Usage: tar c|x|t [-zJjahmvokO] [-f TARFILE] [-C DIR] [-T FILE] [-X FILE] [LONGOPT]... [FILE]...

Create, extract, or list files from a tar file

cCreate
xExtract
tList
-f FILEName of TARFILE ('-' for stdin/out)
-C DIRChange to DIR before operation
-vVerbose
-OExtract to stdout
-mDon't restore mtime
-oDon't restore user:group
-kDon't replace existing files
-z(De)compress using gzip
-J(De)compress using xz
-j(De)compress using bzip2
--lzma(De)compress using lzma
-a(De)compress based on extension
-hFollow symlinks
-T FILEFile with names to include
-X FILEFile with glob patterns to exclude
--exclude PATTERNGlob pattern to exclude
--overwriteReplace existing files
--strip-components NUMNUM of leading components to strip
--no-recursionDon't descend in directories
--numeric-ownerUse numeric user:group
--no-same-permissionsDon't restore access permissions
--to-command COMMANDPipe files to COMMAND
(00.139151) Error (criu/util.c:641): exited, status=1
(00.139164) Error (criu/filesystems.c:426): Can't dump tmpfs content
(00.139239) Unlock network
(00.139242) Running network-unlock scripts
(00.139245) RPC
(00.192473) Unfreezing tasks into 1
(00.192495) Unseizing 29231 into 1
(00.192505) Unseizing 29249 into 1
(00.192527) Error (criu/cr-dump.c:2053): Dumping FAILED.

[adrianreber]

You need gnu tar not busybox tar.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.