cri-o cri checkpoint feature #6749

abdelghanimeliani · 2023-03-21T10:24:57Z

abdelghanimeliani
Mar 21, 2023

hello everyone , in readed this blog post about the new alpha kubelet checkpoint api feature and i wanted to test it but i found that the new image generated from the checkpoint is smaller then first image and when i tied to pull it again i got an error "NO command specified" , mor details are in this issue kubernetes/kubernetes#116744 (comment) .
do you think that the problem is in the cri-o CheckpointContainer implementation or in my envirement maybe ?

Answered by adrianreber

Apr 6, 2023

One thing we did to make this work was disable copy on write on btrfs with chattr +C. Also see https://criu.org/Filesystems_pecularities

View full answer

haircommander · 2023-03-21T14:29:05Z

haircommander
Mar 21, 2023
Maintainer

@adrianreber WDYT

0 replies

adrianreber · 2023-03-21T14:35:39Z

adrianreber
Mar 21, 2023

@abdelghanimeliani please try with the latest version of CRI-O from git.

I think this is related to a fix latest CRI-O git has. A workaround could be to start CRI-O with --drop-infra-ctr=false.

Or start the container with shareProcessNamespace: true. Based on your Kubernetes issue you are using CRI-O 1.25.0 and that had problems restoring containers with pods without infrastructure containers.

0 replies

abdelghanimeliani · 2023-03-22T18:31:16Z

abdelghanimeliani
Mar 22, 2023
Author

@adrianreber , for some reason , im not able to build the package from the source code , is this fix included in version 1.26.1 or 1.26.2 ?

4 replies

adrianreber Mar 22, 2023

Not sure. There is some message about the required go version. Do you have 1.19?

abdelghanimeliani Mar 22, 2023
Author

yes i have 1.20

abdelghanimeliani Mar 22, 2023
Author

i will use 1.19 maybe that's the issue

adrianreber Mar 23, 2023

From the error message it does not sound like you are using 1.20.

abdelghanimeliani · 2023-03-23T12:50:02Z

abdelghanimeliani
Mar 23, 2023
Author

@adrianreber im using the last version of crio from git and go1.19 , i tried to run the pod with shareProcessNamespace: true and to to start CRI-O with --drop-infra-ctr=false , the same problem reported in kubernetes repo is happening

29 replies

adrianreber Mar 23, 2023

You just need to create it.

abdelghanimeliani Mar 23, 2023
Author

@adrianreber , okey here the logs

Error (criu/util.c:641): exited, status=4
Error (criu/util.c:641): exited, status=4
Warn (criu/proc_parse.c:930): Write 4294967295 to /proc/self/loginuid failed: Operation not permitted
Warn (criu/net.c:3427): Unable to get tun network namespace
Warn (criu/sk-unix.c:224): unix: Unable to open a socket file: Operation not permitted
Error (criu/net.c:3711): Unable create a network namespace: Operation not permitted
Warn (criu/net.c:3767): NSID isn't reported for network links
Warn (criu/net.c:3427): Unable to get socket network namespace
Error (criu/net.c:1246): Unexpected error: -1(Operation not permitted)
Warn (criu/kerndat.c:1453): CRIU was built without libnftables support
Error (criu/kerndat.c:1000): Fail to mount tmfps to /tmp/.criu.move_mount_set_group.0g361d: Operation not permitted
Error (criu/kerndat.c:1653): kerndat_has_move_mount_set_group failed when initializing kerndat.
Error (criu/crtools.c:260): Could not initialize kernel features detection.

adrianreber Mar 23, 2023

What kind of system are you using? That looks strange. Is CRI-O running as root?

abdelghanimeliani Mar 23, 2023
Author

@adrianreber im using the "ubuntu/jammy64" vagrant box , and yes it is runing as root , before i was using bionic box but now im using the scripts from this blog post https://martinheinz.dev/blog/85

adrianreber Mar 23, 2023

The CRIU messages are really unusual. It looks like it is not running as root. Can you do a criu check --all -v4.

abdelghanimeliani · 2023-03-23T16:59:32Z

abdelghanimeliani
Mar 23, 2023
Author

@adrianreber
(00.000006) Version: 3.17.1 (gitid 0)
(00.000048) Running on kubemaster Linux 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64
(00.000067) File /run/criu.kdat does not exist
(00.004913) sockets: Probing sock diag modules
(00.005062) sockets: Done probing
(00.007312) Error (criu/util.c:641): exited, status=4
(00.009476) Error (criu/util.c:641): exited, status=4
(00.009561) Pagemap provides flags only
(00.015156) Found anon-shmem device at 1
(00.015217) Hugetlb size 2 Mb is supported but cannot get dev's number
(00.015233) Reset 40278's dirty tracking
(00.015254) Dirty tracking support is OFF
(00.015260) Zero page detection failed, optimization turns off.
(00.015304) Found task size of 7ffffffff000
(00.015338) Warn (criu/proc_parse.c:930): Write 4294967295 to /proc/self/loginuid failed: Operation not permitted
(00.021464) Warn (criu/net.c:3427): Unable to get tun network namespace
(00.021497) Warn (criu/sk-unix.c:224): unix: Unable to open a socket file: Operation not permitted
(00.021791) Error (criu/net.c:3711): Unable create a network namespace: Operation not permitted
(00.022319) Warn (criu/net.c:3767): NSID isn't reported for network links
(00.022407) uffd: Lazy pages are not available: Operation not permitted
(00.022411) Lazy pages are not permitted
(00.023236) vdso: Parsing at 7ffdd77fd000 7ffdd77ff000
(00.023240) vdso: PT_LOAD p_vaddr: 0
(00.023243) vdso: DT_HASH: 120
(00.023245) vdso: DT_STRTAB: 300
(00.023248) vdso: DT_SYMTAB: 1c8
(00.023250) vdso: DT_STRSZ: 8b
(00.023252) vdso: DT_SYMENT: 18
(00.023254) vdso: nbucket 3 nchain d bucket 7ffdd77fd128 chain 7ffdd77fd134
(00.023258) vdso: rt [vdso] 7ffdd77fd000-7ffdd77ff000 [vvar] 7ffdd77f9000-7ffdd77fd000
(00.023767) vdso: Parsing at 7fe7c2062000 7fe7c2064000
(00.023776) vdso: PT_LOAD p_vaddr: 0
(00.023779) vdso: DT_HASH: b4
(00.023781) vdso: DT_STRTAB: 1f0
(00.023783) vdso: DT_SYMTAB: 140
(00.023786) vdso: DT_STRSZ: c0
(00.023788) vdso: DT_SYMENT: 10
(00.023790) vdso: nbucket 3 nchain b bucket 7fe7c20620bc chain 7fe7c20620c8
(00.023793) vdso: compat [vdso] 7ffdd7801000-7ffdd7803000 [vvar] 7ffdd77fd000-7ffdd7801000
(00.024302) Warn (criu/net.c:3427): Unable to get socket network namespace
(00.024343) cpu: x86_family 6 x86_vendor_id GenuineIntel x86_model_id Intel(R) Core(TM) i5-8365U CPU @ 1.60GHz
(00.024354) cpu: fpu: xfeatures_mask 0x5 xsave_size 832 xsave_size_max 832 xsaves_size 960
(00.024359) cpu: fpu: x87 floating point registers xstate_offsets 0 / 0 xstate_sizes 160 / 160
(00.024363) cpu: fpu: AVX registers xstate_offsets 576 / 576 xstate_sizes 256 / 256
(00.024462) Error (criu/net.c:1246): Unexpected error: -1(Operation not permitted)
(00.024522) Warn (criu/kerndat.c:1453): CRIU was built without libnftables support
(00.024604) Error (criu/kerndat.c:1000): Fail to mount tmfps to /tmp/.criu.move_mount_set_group.jaVOGx: Operation not permitted
(00.024638) Error (criu/kerndat.c:1653): kerndat_has_move_mount_set_group failed when initializing kerndat.
(00.024694) Found mmap_min_addr 0x10000
(00.024704) files stat: fs/nr_open 1048576
(00.024706) Error (criu/crtools.c:260): Could not initialize kernel features detection.

15 replies

abdelghanimeliani Mar 29, 2023
Author

@adrianreber

what about this

Warn  (criu/crtools.c:303): Using -t with criu restore is obsoleted
Error (criu/protobuf.c:72): Unexpected EOF on (empty-image)

adrianreber Mar 29, 2023

That also looks like CRIU has been started from the command-line and not from CRI-O.

abdelghanimeliani Mar 30, 2023
Author

@adrianreber i tested it today whit fedora37 , i got the same error msg="criu failed: type NOTIFY errno 0\nlog file: /run/containers/storage/overlay-containers/counter/userdata/restore.log" , the kernel version is 6.0.7-301.fc37.x86_64 and everything is running as root , can you tell me what system you are using and how did you setup your envirement ?

adrianreber Mar 30, 2023

Fedora 37 and compiled CRI-O from latest git

adrianreber Mar 30, 2023

Can you try to reach out to me on slack or something like that.

adrianreber · 2023-04-06T07:00:15Z

adrianreber
Apr 6, 2023

One thing we did to make this work was disable copy on write on btrfs with chattr +C. Also see https://criu.org/Filesystems_pecularities

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cri-o cri checkpoint feature #6749

{{title}}

Replies: 6 comments 48 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

cri-o cri checkpoint feature #6749

Replies: 6 comments · 48 replies

haircommander Mar 21, 2023 Maintainer

abdelghanimeliani Mar 22, 2023 Author

abdelghanimeliani Mar 22, 2023 Author

abdelghanimeliani Mar 22, 2023 Author

abdelghanimeliani Mar 23, 2023 Author

abdelghanimeliani Mar 23, 2023 Author

abdelghanimeliani Mar 23, 2023 Author

abdelghanimeliani Mar 23, 2023 Author

abdelghanimeliani Mar 29, 2023 Author

abdelghanimeliani Mar 30, 2023 Author

Replies: 6 comments 48 replies

haircommander
Mar 21, 2023
Maintainer

abdelghanimeliani
Mar 22, 2023
Author

abdelghanimeliani Mar 22, 2023
Author

abdelghanimeliani Mar 22, 2023
Author

abdelghanimeliani
Mar 23, 2023
Author

abdelghanimeliani Mar 23, 2023
Author

abdelghanimeliani Mar 23, 2023
Author

abdelghanimeliani
Mar 23, 2023
Author

abdelghanimeliani Mar 29, 2023
Author

abdelghanimeliani Mar 30, 2023
Author