Restoring Docker container with docker_cr.sh failed #3

kimh · 2015-05-16T13:26:13Z

I tried to ckeckpoint/restore a container with docker_cr.sh. Checkpoint succeeded but I couldn't make restarting works.

Here is my env.

Ubuntu: 14.04 (vagrant)

CRIU: Compiled from source at xemul@61c1936

Docker: Compiled from source at https://github.com/SaiedKazemi/docker/commit/62ce554ce42e3ad0f5c4c69896685903ecbd1ffc

I could checkpoint/restore a simple bash process with compiled criu, so I believe installation is done properly.

Here is what I did.

# Run a docker container in the example
root@vagrant-ubuntu-trusty:~# export CID=$(docker run -d busybox:latest /bin/sh -c 'i=0; while true; do echo $i >> /foo; i=$(expr $i + 1
); sleep 3; done')

# Make sure it's running
root@vagrant-ubuntu-trusty:~# docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS               NAMES
cb8fd3d21034        busybox:latest      /bin/sh -c 'i=0; whi   3 seconds ago       Up 2 seconds                            berserk_rosalind

# Checkpoint container succeeds
root@vagrant-ubuntu-trusty:~/criu/contrib# ./docker_cr.sh -c -v $CID
docker binary: docker
dockerinit binary: /var/lib/docker/init/dockerinit-1.0.1
criu binary: criu
image directory: /var/lib/docker/criu_img/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c
container root directory: /var/lib/docker/aufs/mnt/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c

criu dump -v4 -D /var/lib/docker/criu_img/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c -o dump.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/etc/resolv.conf --ext-mount-map /etc/hosts:/etc/hosts --ext-mount-map /etc/hostname:/etc/hostname --ext-mount-map /.dockerinit:/.dockerinit -t 1426 --root /var/lib/docker/aufs/mnt/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c


dump successful
(00.214765) Dumping finished successfully

# Restoring doesn't succeed, though
root@vagrant-ubuntu-trusty:~/criu/contrib# ./docker_cr.sh -r -v $CID
docker binary: docker
dockerinit binary: /var/lib/docker/init/dockerinit-1.0.1
criu binary: criu
image directory: /var/lib/docker/criu_img/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c
container root directory: /var/lib/docker/aufs/mnt/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c

mount -t aufs -o
/var/lib/docker/aufs/diff/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c
/var/lib/docker/aufs/diff/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c-init
/var/lib/docker/aufs/diff/8c2e06607696bd4afb3d03b687e361cc43cf8ec1a4a725bc96e39f05ba97dd55
/var/lib/docker/aufs/diff/6ce2e90b0bc7224de3db1f0d646fe8e2c4dd37f1793928287f6074bc451a57ea
/var/lib/docker/aufs/diff/cf2616975b4a3cba083ca99bc3f0bf25f5f528c3c52be1596b30f60b0b1c37ff
none
/var/lib/docker/aufs/mnt/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c

criu restore -v4 -D /var/lib/docker/criu_img/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c -o restore.log --manage-cgroups --evasive-devices --ext-mount-map /etc/resolv.conf:/var/lib/docker/containers/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c/resolv.conf --ext-mount-map /etc/hosts:/var/lib/docker/containers/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c/hosts --ext-mount-map /etc/hostname:/var/lib/docker/containers/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c/hostname --ext-mount-map /.dockerinit:/var/lib/docker/init/dockerinit-1.0.1 -d --root /var/lib/docker/aufs/mnt/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c --pidfile /var/lib/docker/criu_img/cb8fd3d210342905c03f762bf8fccff366ef63327bf2429ead916683daa5122c/restore.pid


restore failed
(00.106843) Error (cr-restore.c:1221): 1622 exited, status=1
(00.119208) Error (cr-restore.c:1895): Restoring FAILED.

Any ideas how what's wrong?

The text was updated successfully, but these errors were encountered:

xemul · 2015-05-18T08:08:26Z

Hm... @SaiedKazemi, can you help us, please :)
@kimh , I guess a full restore log would be required.

kimh · 2015-05-18T09:15:42Z

@xemul This is the all log that I got. How can I take the full restore log?

avagin · 2015-05-18T10:58:38Z

@boucher is integrating CRIU into Docker https://github.com/boucher/docker/tree/boucher-cr, so it's better to use this tree and share your experience.

kimh · 2015-05-18T12:12:34Z

I compiled docker from https://github.com/boucher/docker/tree/boucher-cr, but result is the same. Resuming fails with the same error message.

avagin · 2015-05-18T12:23:40Z

@kimh I need a fill restore log to investigate this issue. Could you find and attach the restore.log file in a image directory?

kimh · 2015-05-18T12:31:27Z

@avagin Uploaded to https://gist.github.com/kimh/14eef27af10b320d6bdd

avagin · 2015-05-18T13:17:45Z

@kimh, could you compile criu from this repo avagin@f64c0f1 and attach the restore.log again. Thank you for the help.

kimh · 2015-05-18T13:27:57Z

@avagin Uploaded to https://gist.github.com/kimh/fb6f8ab3eae5446bbeb7

I don't see additional logging that you added, but I'm sure that I compiled from your branch.

root@vagrant-ubuntu-trusty:~/criu/contrib# git branch
* issue-3
  master

Let me know if you need more info.

avagin · 2015-05-18T13:35:17Z

Could you do one more attempt with this tree avagin@e11f048?

kimh · 2015-05-18T13:44:52Z

@avagin Uploaded to https://gist.github.com/kimh/dc6da8dc1fdac718235d

avagin · 2015-05-18T14:09:42Z

I don't see debug messages. Could you check that you use the right version of criu? Maybe you forgot to compile it.

kimh · 2015-05-18T14:20:07Z

@avagin Oops, I use docker_cr.sh but it uses globally installed criu binary. I did this to use compiled criu.

 CRIU_BINARY=/home/vagrant/criu/criu ./docker_cr.sh -r -v $CID

and uploaded restore.log to https://gist.github.com/kimh/03f5f01e61b1e9616c8a

avagin · 2015-05-18T14:30:23Z

(00.089080) 56: Error (lsm.c:202): Unknown LSM type: 0
Cc: @tych0

avagin · 2015-05-18T14:58:14Z

Could you do one more time with this avagin@51272a9?

kimh · 2015-05-18T15:13:17Z

@avagin Uploaded to https://gist.github.com/kimh/4f2aa1eed6ffea2f186e

avagin · 2015-05-18T15:27:02Z

Pls, try out this avagin@9fc4975. I think I've fixed this issue there. Thanks.

kimh · 2015-05-18T15:42:23Z

@avagin Worked!

After the successful resuming, docker ps doesn't show the resumed container. This is becausedocker_cr.sh uses external C/R, not native?

Thank you for your support.

tych0 · 2015-05-18T17:11:55Z

@avagin Looks good, does that mean we should get rid of the calls to host_lsm_type() in lsm.h?

avagin · 2015-05-18T17:14:38Z

@tych0, I'm going to send a patch soon.

boucher · 2015-05-18T17:21:48Z

Yes, docker_cr.sh can't update the daemon's internal state, so docker won't
know about the container any longer. If you're able to get my branch
working, you should see the restored container in 'docker ps'

On Mon, May 18, 2015 at 8:42 AM, Kim, Hirokuni notifications@github.com
wrote:

@avagin https://github.com/avagin Worked!

After the successful resuming, docker ps doesn't show the resumed
container. This is becausedocker_cr.sh uses external C/R, not native?

Thank you for your support.

—
Reply to this email directly or view it on GitHub
https://github.com/xemul/criu/issues/3#issuecomment-103108175.

SaiedKazemi · 2015-05-18T19:25:55Z

@xemul Sorry for the delay as I am on the last leg of my return flight... As @avagin and @boucher have mentioned, we should use native Docker checkpoint and restore from this point on. The helper script docker_cr.sh served its main purpose of showing proof of concept for external C/R with CRIU 1.3. I think we should deprecate it in 1.6.

kimh · 2015-05-19T14:23:52Z

I'm closing this issue since docker_cr.sh now works.

If you call clone directly you are responsible for setting up the TLS area yourself. $ abrt-cli ls | grep different_creds | wc -l 39 $ gdb -c /var/spool/abrt/ccpp-2015-07-24-10\:21\:14-8014/coredump different_creds Core was generated by `./different_creds --pidfile=different_creds.pid --outfile=different_creds.out'. Program terminated with signal SIGILL, Illegal instruction. #0 0x00007f86e2d8c7d9 in _dl_x86_64_restore_sse () from /lib64/ld-linux-x86-64.so.2 Missing separate debuginfos, use: dnf debuginfo-install glibc-2.21-7.fc22.x86_64 libattr-2.4.47-9.fc22.x86_64 libcap-2.24-7.fc22.x86_64 (gdb) bt #0 0x00007f86e2d8c7d9 in _dl_x86_64_restore_sse () from /lib64/ld-linux-x86-64.so.2 #1 0x00007f86e2d84add in _dl_fixup () from /lib64/ld-linux-x86-64.so.2 #2 0x00007f86e2d8bbc0 in _dl_runtime_resolve () from /lib64/ld-linux-x86-64.so.2 #3 0x0000000000402da3 in sys_futex (val3=0, uaddr2=0x0, timeout=0x0, val=0, op=0, uaddr=0x6063f0 <sig_received>) at lock.h:29 #4 futex_wait_while (f=0x6063f0 <sig_received>, v=0) at lock.h:121 #5 test_waitsig () at test.c:367 #6 0x0000000000401c4b in main (argc=<optimized out>, argv=0x7ffce16432f8) at different_creds.c:82 Reported-by: Mr Jenkins Cc: Tycho Andersen <tycho.andersen@canonical.com> Signed-off-by: Andrew Vagin <avagin@openvz.org> Acked-by: Tycho Andersen <tycho.andersen@canonical.com> Signed-off-by: Pavel Emelyanov <xemul@parallels.com>

It can be dead-lokced: #0 0x00007fafbf49f6ac in __lll_lock_wait_private () from /lib64/libc.so.6 #1 0x00007fafbf44af1c in _L_lock_2460 () from /lib64/libc.so.6 #2 0x00007fafbf44ad57 in __tz_convert () from /lib64/libc.so.6 checkpoint-restore#3 0x00000000004022e2 in test_msg (format=0x404508 "Receive signal %d\n") at msg.c:51 checkpoint-restore#4 <signal handler called> checkpoint-restore#5 0x00007fafbf3f2483 in __GI__IO_vfscanf () from /lib64/libc.so.6 checkpoint-restore#6 0x00007fafbf408f27 in vsscanf () from /lib64/libc.so.6 checkpoint-restore#7 0x00007fafbf4032f7 in sscanf () from /lib64/libc.so.6 checkpoint-restore#8 0x00007fafbf449ba6 in __tzset_parse_tz () from /lib64/libc.so.6 checkpoint-restore#9 0x00007fafbf44c4cb in __tzfile_compute () from /lib64/libc.so.6 checkpoint-restore#10 0x00007fafbf44ae17 in __tz_convert () from /lib64/libc.so.6 checkpoint-restore#11 0x00000000004022e2 in test_msg (format=format@entry=0x40458c "PASS\n") at msg.c:51 checkpoint-restore#12 0x0000000000401ceb in main (argc=<optimized out>, argv=<optimized out>) at ptrace_sig.c:172 https://jira.sw.ru/browse/PSBM-47772 Signed-off-by: Andrey Vagin <avagin@openvz.org> Signed-off-by: Andrew Vagin <avagin@virtuozzo.com> Tested-by: Cyrill Gorcunov <gorcunov@openvz.org>

It can be dead-lokced: #0 0x00007fafbf49f6ac in __lll_lock_wait_private () from /lib64/libc.so.6 #1 0x00007fafbf44af1c in _L_lock_2460 () from /lib64/libc.so.6 #2 0x00007fafbf44ad57 in __tz_convert () from /lib64/libc.so.6 #3 0x00000000004022e2 in test_msg (format=0x404508 "Receive signal %d\n") at msg.c:51 #4 <signal handler called> #5 0x00007fafbf3f2483 in __GI__IO_vfscanf () from /lib64/libc.so.6 #6 0x00007fafbf408f27 in vsscanf () from /lib64/libc.so.6 #7 0x00007fafbf4032f7 in sscanf () from /lib64/libc.so.6 #8 0x00007fafbf449ba6 in __tzset_parse_tz () from /lib64/libc.so.6 #9 0x00007fafbf44c4cb in __tzfile_compute () from /lib64/libc.so.6 #10 0x00007fafbf44ae17 in __tz_convert () from /lib64/libc.so.6 #11 0x00000000004022e2 in test_msg (format=format@entry=0x40458c "PASS\n") at msg.c:51 #12 0x0000000000401ceb in main (argc=<optimized out>, argv=<optimized out>) at ptrace_sig.c:172 Signed-off-by: Andrey Vagin <avagin@openvz.org> Signed-off-by: Andrew Vagin <avagin@virtuozzo.com> Tested-by: Cyrill Gorcunov <gorcunov@openvz.org> Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>

…VZ.ORG/criu:overmounts to vz7-u3 * commit '619c09da6bd96be7be2f3025a7f33b51adcc9739': zdtm: add a test to check a mount tree with overmounts mount: clone a mount namespace to open a detached mount mnt: try to split a mount tree to restore over-mounted mounts mnt: rename roots_mp into root_yard_mp and make it global mnt: add a sub-id to print_ns_root mnt: clean up mnt: add a function to check whether a mount overmounts something

Ghost file entry used right after it has been freed: ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000dc50 READ of size 4 at 0x60700000dc50 thread T0 #0 0x46e819 in open_remap_ghost criu/files-reg.c:312 #1 0x46e819 in prepare_one_remap criu/files-reg.c:461 #2 0x46e819 in prepare_remaps criu/files-reg.c:507 checkpoint-restore#3 0x45af00 in root_prepare_shared criu/cr-restore.c:235 checkpoint-restore#4 0x45af00 in restore_task_with_children criu/cr-restore.c:1421 checkpoint-restore#5 0x7efc71e85f0c in clone (/lib64/libc.so.6+0xe7f0c) 0x60700000dc50 is located 32 bytes inside of 80-byte region [0x60700000dc30,0x60700000dc80) freed by thread T0 here: #0 0x7efc7305184a in __interceptor_free (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x9884a) #1 0x46e4df in open_remap_ghost criu/files-reg.c:309 #2 0x46e4df in prepare_one_remap criu/files-reg.c:461 checkpoint-restore#3 0x46e4df in prepare_remaps criu/files-reg.c:507 previously allocated by thread T0 here: #0 0x7efc73051b82 in malloc (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x98b82) #1 0x7efc7277a8ea in protobuf_c_message_unpack (/usr/lib64/libprotobuf-c.so.1+0x48ea) #2 0xd528232002838017 (<unknown module>) Just move freeing after the last 'gfe' usage to fix this. Fixes: d0097b2 ("files: Support ghost directories restore") Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com>

'info' array is off-by-one, nla_parse_nested() requires destination array (i.e. 'info') to have maxtype+1 (i.e. IFLA_INFO_MAX+1) elements: ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffef823e3f8 WRITE of size 48 at 0x7ffef823e3f8 thread T0 #0 0x7f9ab7a3915b in __asan_memset (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x8d15b) #1 0x7f9ab6d4e553 in nla_parse (/usr/lib64/libnl-3.so.200+0xa553) #2 0x4acfb7 in dump_one_netdev criu/net.c:445 checkpoint-restore#3 0x4adb60 in dump_one_ethernet criu/net.c:594 checkpoint-restore#4 0x4adb60 in dump_one_link criu/net.c:665 checkpoint-restore#5 0x48af69 in nlmsg_receive criu/libnetlink.c:45 checkpoint-restore#6 0x48af69 in do_rtnl_req criu/libnetlink.c:119 checkpoint-restore#7 0x4b0e86 in dump_links criu/net.c:878 checkpoint-restore#8 0x4b0e86 in dump_net_ns criu/net.c:1651 checkpoint-restore#9 0x4a760d in do_dump_namespaces criu/namespaces.c:985 checkpoint-restore#10 0x4a760d in dump_namespaces criu/namespaces.c:1045 checkpoint-restore#11 0x451ef7 in cr_dump_tasks criu/cr-dump.c:1799 checkpoint-restore#12 0x424588 in main criu/crtools.c:736 checkpoint-restore#13 0x7f9ab67b171f in __libc_start_main (/lib64/libc.so.6+0x2071f) checkpoint-restore#14 0x4253d8 in _start (/criu/criu/criu+0x4253d8) Address 0x7ffef823e3f8 is located in stack of thread T0 at offset 264 in frame #0 0x4ac9ef in dump_one_netdev criu/net.c:364 This frame has 5 object(s): [32, 168) 'netdev' [224, 264) 'info' <== Memory access at offset 264 overflows this variable [320, 1040) 'req' [1088, 3368) 'path' [3424, 3625) 'stable_secret' Increase 'info' size to fix this. Fixes: b705dcc ("net: pass the struct nlattrs to dump() functions") Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com>

Ghost file entry used right after it has been freed: ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000dc50 READ of size 4 at 0x60700000dc50 thread T0 #0 0x46e819 in open_remap_ghost criu/files-reg.c:312 #1 0x46e819 in prepare_one_remap criu/files-reg.c:461 #2 0x46e819 in prepare_remaps criu/files-reg.c:507 #3 0x45af00 in root_prepare_shared criu/cr-restore.c:235 #4 0x45af00 in restore_task_with_children criu/cr-restore.c:1421 #5 0x7efc71e85f0c in clone (/lib64/libc.so.6+0xe7f0c) 0x60700000dc50 is located 32 bytes inside of 80-byte region [0x60700000dc30,0x60700000dc80) freed by thread T0 here: #0 0x7efc7305184a in __interceptor_free (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x9884a) #1 0x46e4df in open_remap_ghost criu/files-reg.c:309 #2 0x46e4df in prepare_one_remap criu/files-reg.c:461 #3 0x46e4df in prepare_remaps criu/files-reg.c:507 previously allocated by thread T0 here: #0 0x7efc73051b82 in malloc (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x98b82) #1 0x7efc7277a8ea in protobuf_c_message_unpack (/usr/lib64/libprotobuf-c.so.1+0x48ea) #2 0xd528232002838017 (<unknown module>) Just move freeing after the last 'gfe' usage to fix this. Fixes: d0097b2 ("files: Support ghost directories restore") travis-ci: success for files-reg: fix use-after-free in open_remap_ghost() Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>

'info' array is off-by-one, nla_parse_nested() requires destination array (i.e. 'info') to have maxtype+1 (i.e. IFLA_INFO_MAX+1) elements: ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffef823e3f8 WRITE of size 48 at 0x7ffef823e3f8 thread T0 #0 0x7f9ab7a3915b in __asan_memset (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x8d15b) #1 0x7f9ab6d4e553 in nla_parse (/usr/lib64/libnl-3.so.200+0xa553) #2 0x4acfb7 in dump_one_netdev criu/net.c:445 #3 0x4adb60 in dump_one_ethernet criu/net.c:594 #4 0x4adb60 in dump_one_link criu/net.c:665 #5 0x48af69 in nlmsg_receive criu/libnetlink.c:45 #6 0x48af69 in do_rtnl_req criu/libnetlink.c:119 #7 0x4b0e86 in dump_links criu/net.c:878 #8 0x4b0e86 in dump_net_ns criu/net.c:1651 #9 0x4a760d in do_dump_namespaces criu/namespaces.c:985 #10 0x4a760d in dump_namespaces criu/namespaces.c:1045 #11 0x451ef7 in cr_dump_tasks criu/cr-dump.c:1799 #12 0x424588 in main criu/crtools.c:736 #13 0x7f9ab67b171f in __libc_start_main (/lib64/libc.so.6+0x2071f) #14 0x4253d8 in _start (/criu/criu/criu+0x4253d8) Address 0x7ffef823e3f8 is located in stack of thread T0 at offset 264 in frame #0 0x4ac9ef in dump_one_netdev criu/net.c:364 This frame has 5 object(s): [32, 168) 'netdev' [224, 264) 'info' <== Memory access at offset 264 overflows this variable [320, 1040) 'req' [1088, 3368) 'path' [3424, 3625) 'stable_secret' Increase 'info' size to fix this. Fixes: b705dcc ("net: pass the struct nlattrs to dump() functions") travis-ci: success for net: fix stack out-of-bounds access in dump_one_netdev() Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Acked-by: Cyrill Gorcunov <gorcunov@openvz.org> Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>

Ghost file entry used right after it has been freed: ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000dc50 READ of size 4 at 0x60700000dc50 thread T0 #0 0x46e819 in open_remap_ghost criu/files-reg.c:312 #1 0x46e819 in prepare_one_remap criu/files-reg.c:461 #2 0x46e819 in prepare_remaps criu/files-reg.c:507 #3 0x45af00 in root_prepare_shared criu/cr-restore.c:235 #4 0x45af00 in restore_task_with_children criu/cr-restore.c:1421 #5 0x7efc71e85f0c in clone (/lib64/libc.so.6+0xe7f0c) 0x60700000dc50 is located 32 bytes inside of 80-byte region [0x60700000dc30,0x60700000dc80) freed by thread T0 here: #0 0x7efc7305184a in __interceptor_free (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x9884a) #1 0x46e4df in open_remap_ghost criu/files-reg.c:309 #2 0x46e4df in prepare_one_remap criu/files-reg.c:461 #3 0x46e4df in prepare_remaps criu/files-reg.c:507 previously allocated by thread T0 here: #0 0x7efc73051b82 in malloc (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x98b82) #1 0x7efc7277a8ea in protobuf_c_message_unpack (/usr/lib64/libprotobuf-c.so.1+0x48ea) #2 0xd528232002838017 (<unknown module>) Just move freeing after the last 'gfe' usage to fix this. Fixes: d0097b2 ("files: Support ghost directories restore") travis-ci: success for files-reg: fix use-after-free in open_remap_ghost() Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>

'info' array is off-by-one, nla_parse_nested() requires destination array (i.e. 'info') to have maxtype+1 (i.e. IFLA_INFO_MAX+1) elements: ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffef823e3f8 WRITE of size 48 at 0x7ffef823e3f8 thread T0 #0 0x7f9ab7a3915b in __asan_memset (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x8d15b) #1 0x7f9ab6d4e553 in nla_parse (/usr/lib64/libnl-3.so.200+0xa553) #2 0x4acfb7 in dump_one_netdev criu/net.c:445 #3 0x4adb60 in dump_one_ethernet criu/net.c:594 #4 0x4adb60 in dump_one_link criu/net.c:665 #5 0x48af69 in nlmsg_receive criu/libnetlink.c:45 #6 0x48af69 in do_rtnl_req criu/libnetlink.c:119 #7 0x4b0e86 in dump_links criu/net.c:878 #8 0x4b0e86 in dump_net_ns criu/net.c:1651 #9 0x4a760d in do_dump_namespaces criu/namespaces.c:985 #10 0x4a760d in dump_namespaces criu/namespaces.c:1045 #11 0x451ef7 in cr_dump_tasks criu/cr-dump.c:1799 #12 0x424588 in main criu/crtools.c:736 #13 0x7f9ab67b171f in __libc_start_main (/lib64/libc.so.6+0x2071f) #14 0x4253d8 in _start (/criu/criu/criu+0x4253d8) Address 0x7ffef823e3f8 is located in stack of thread T0 at offset 264 in frame #0 0x4ac9ef in dump_one_netdev criu/net.c:364 This frame has 5 object(s): [32, 168) 'netdev' [224, 264) 'info' <== Memory access at offset 264 overflows this variable [320, 1040) 'req' [1088, 3368) 'path' [3424, 3625) 'stable_secret' Increase 'info' size to fix this. Fixes: b705dcc ("net: pass the struct nlattrs to dump() functions") travis-ci: success for net: fix stack out-of-bounds access in dump_one_netdev() Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Acked-by: Cyrill Gorcunov <gorcunov@openvz.org> Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>

Ghost file entry used right after it has been freed: ERROR: AddressSanitizer: heap-use-after-free on address 0x60700000dc50 READ of size 4 at 0x60700000dc50 thread T0 #0 0x46e819 in open_remap_ghost criu/files-reg.c:312 #1 0x46e819 in prepare_one_remap criu/files-reg.c:461 #2 0x46e819 in prepare_remaps criu/files-reg.c:507 #3 0x45af00 in root_prepare_shared criu/cr-restore.c:235 #4 0x45af00 in restore_task_with_children criu/cr-restore.c:1421 #5 0x7efc71e85f0c in clone (/lib64/libc.so.6+0xe7f0c) 0x60700000dc50 is located 32 bytes inside of 80-byte region [0x60700000dc30,0x60700000dc80) freed by thread T0 here: #0 0x7efc7305184a in __interceptor_free (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x9884a) #1 0x46e4df in open_remap_ghost criu/files-reg.c:309 #2 0x46e4df in prepare_one_remap criu/files-reg.c:461 #3 0x46e4df in prepare_remaps criu/files-reg.c:507 previously allocated by thread T0 here: #0 0x7efc73051b82 in malloc (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x98b82) #1 0x7efc7277a8ea in protobuf_c_message_unpack (/usr/lib64/libprotobuf-c.so.1+0x48ea) #2 0xd528232002838017 (<unknown module>) Just move freeing after the last 'gfe' usage to fix this. Fixes: d0097b2 ("files: Support ghost directories restore") travis-ci: success for files-reg: fix use-after-free in open_remap_ghost() Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>

'info' array is off-by-one, nla_parse_nested() requires destination array (i.e. 'info') to have maxtype+1 (i.e. IFLA_INFO_MAX+1) elements: ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffef823e3f8 WRITE of size 48 at 0x7ffef823e3f8 thread T0 #0 0x7f9ab7a3915b in __asan_memset (/usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libasan.so.2+0x8d15b) #1 0x7f9ab6d4e553 in nla_parse (/usr/lib64/libnl-3.so.200+0xa553) #2 0x4acfb7 in dump_one_netdev criu/net.c:445 #3 0x4adb60 in dump_one_ethernet criu/net.c:594 #4 0x4adb60 in dump_one_link criu/net.c:665 #5 0x48af69 in nlmsg_receive criu/libnetlink.c:45 #6 0x48af69 in do_rtnl_req criu/libnetlink.c:119 #7 0x4b0e86 in dump_links criu/net.c:878 #8 0x4b0e86 in dump_net_ns criu/net.c:1651 #9 0x4a760d in do_dump_namespaces criu/namespaces.c:985 #10 0x4a760d in dump_namespaces criu/namespaces.c:1045 #11 0x451ef7 in cr_dump_tasks criu/cr-dump.c:1799 #12 0x424588 in main criu/crtools.c:736 #13 0x7f9ab67b171f in __libc_start_main (/lib64/libc.so.6+0x2071f) #14 0x4253d8 in _start (/criu/criu/criu+0x4253d8) Address 0x7ffef823e3f8 is located in stack of thread T0 at offset 264 in frame #0 0x4ac9ef in dump_one_netdev criu/net.c:364 This frame has 5 object(s): [32, 168) 'netdev' [224, 264) 'info' <== Memory access at offset 264 overflows this variable [320, 1040) 'req' [1088, 3368) 'path' [3424, 3625) 'stable_secret' Increase 'info' size to fix this. Fixes: b705dcc ("net: pass the struct nlattrs to dump() functions") travis-ci: success for net: fix stack out-of-bounds access in dump_one_netdev() Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Acked-by: Cyrill Gorcunov <gorcunov@openvz.org> Signed-off-by: Pavel Emelyanov <xemul@virtuozzo.com>

==30==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000e3ca at pc 0x7f34144b6be1 bp 0x7ffee7b6bb20 sp 0x7ffee7b6b298 READ of size 26 at 0x60300000e3ca thread T0 #0 0x7f34144b6be0 (/lib64/libasan.so.3+0x8dbe0) #1 0x7f34144b8e4d in __interceptor_vsnprintf (/lib64/libasan.so.3+0x8fe4d) #2 0x4966cb in vprint_on_level criu/log.c:228 checkpoint-restore#3 0x496b64 in print_on_level criu/log.c:249 checkpoint-restore#4 0x505c94 in collect_one_unixsk criu/sk-unix.c:1401 checkpoint-restore#5 0x4e7ae3 in collect_image criu/protobuf.c:213 checkpoint-restore#6 0x462c5c in root_prepare_shared criu/cr-restore.c:247 checkpoint-restore#7 0x462c5c in restore_task_with_children criu/cr-restore.c:1420 checkpoint-restore#8 0x7f34132d70ec in __clone (/lib64/libc.so.6+0x1030ec) 0x60300000e3ca is located 0 bytes to the right of 26-byte region [0x60300000e3b0,0x60300000e3ca) allocated by thread T0 here: #0 0x7f34144efe70 in malloc (/lib64/libasan.so.3+0xc6e70) #1 0x7f3413bdb021 (/lib64/libprotobuf-c.so.1+0x6021) Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>

In this patch, we replace all zero characters to '@'. ==30==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000e3ca at pc 0x7f34144b6be1 bp 0x7ffee7b6bb20 sp 0x7ffee7b6b298 READ of size 26 at 0x60300000e3ca thread T0 #0 0x7f34144b6be0 (/lib64/libasan.so.3+0x8dbe0) #1 0x7f34144b8e4d in __interceptor_vsnprintf (/lib64/libasan.so.3+0x8fe4d) #2 0x4966cb in vprint_on_level criu/log.c:228 checkpoint-restore#3 0x496b64 in print_on_level criu/log.c:249 checkpoint-restore#4 0x505c94 in collect_one_unixsk criu/sk-unix.c:1401 checkpoint-restore#5 0x4e7ae3 in collect_image criu/protobuf.c:213 checkpoint-restore#6 0x462c5c in root_prepare_shared criu/cr-restore.c:247 checkpoint-restore#7 0x462c5c in restore_task_with_children criu/cr-restore.c:1420 checkpoint-restore#8 0x7f34132d70ec in __clone (/lib64/libc.so.6+0x1030ec) 0x60300000e3ca is located 0 bytes to the right of 26-byte region [0x60300000e3b0,0x60300000e3ca) allocated by thread T0 here: #0 0x7f34144efe70 in malloc (/lib64/libasan.so.3+0xc6e70) #1 0x7f3413bdb021 (/lib64/libprotobuf-c.so.1+0x6021) Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>

In this patch, we replace all zero characters to '@'. ==30==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000e3ca at pc 0x7f34144b6be1 bp 0x7ffee7b6bb20 sp 0x7ffee7b6b298 READ of size 26 at 0x60300000e3ca thread T0 #0 0x7f34144b6be0 (/lib64/libasan.so.3+0x8dbe0) #1 0x7f34144b8e4d in __interceptor_vsnprintf (/lib64/libasan.so.3+0x8fe4d) #2 0x4966cb in vprint_on_level criu/log.c:228 #3 0x496b64 in print_on_level criu/log.c:249 #4 0x505c94 in collect_one_unixsk criu/sk-unix.c:1401 #5 0x4e7ae3 in collect_image criu/protobuf.c:213 #6 0x462c5c in root_prepare_shared criu/cr-restore.c:247 #7 0x462c5c in restore_task_with_children criu/cr-restore.c:1420 #8 0x7f34132d70ec in __clone (/lib64/libc.so.6+0x1030ec) 0x60300000e3ca is located 0 bytes to the right of 26-byte region [0x60300000e3b0,0x60300000e3ca) allocated by thread T0 here: #0 0x7f34144efe70 in malloc (/lib64/libasan.so.3+0xc6e70) #1 0x7f3413bdb021 (/lib64/libprotobuf-c.so.1+0x6021) Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>

In this patch, we replace all zero characters to '@'. ==30==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60300000e3ca at pc 0x7f34144b6be1 bp 0x7ffee7b6bb20 sp 0x7ffee7b6b298 READ of size 26 at 0x60300000e3ca thread T0 #0 0x7f34144b6be0 (/lib64/libasan.so.3+0x8dbe0) checkpoint-restore#1 0x7f34144b8e4d in __interceptor_vsnprintf (/lib64/libasan.so.3+0x8fe4d) checkpoint-restore#2 0x4966cb in vprint_on_level criu/log.c:228 checkpoint-restore#3 0x496b64 in print_on_level criu/log.c:249 checkpoint-restore#4 0x505c94 in collect_one_unixsk criu/sk-unix.c:1401 checkpoint-restore#5 0x4e7ae3 in collect_image criu/protobuf.c:213 checkpoint-restore#6 0x462c5c in root_prepare_shared criu/cr-restore.c:247 checkpoint-restore#7 0x462c5c in restore_task_with_children criu/cr-restore.c:1420 checkpoint-restore#8 0x7f34132d70ec in __clone (/lib64/libc.so.6+0x1030ec) 0x60300000e3ca is located 0 bytes to the right of 26-byte region [0x60300000e3b0,0x60300000e3ca) allocated by thread T0 here: #0 0x7f34144efe70 in malloc (/lib64/libasan.so.3+0xc6e70) checkpoint-restore#1 0x7f3413bdb021 (/lib64/libprotobuf-c.so.1+0x6021) Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>

==36==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000001c at pc 0x7fb26c88d5f9 bp 0x7ffc15087d40 sp 0x7ffc150874d0 WRITE of size 13 at 0x60200000001c thread T0 #0 0x7fb26c88d5f8 in vsprintf (/lib64/libasan.so.4+0x9e5f8) #1 0x7fb26c88d986 in __interceptor_sprintf (/lib64/libasan.so.4+0x9e986) #2 0x402453 in main /root/git/main/criu/test/zdtm/static/chroot.c:68 checkpoint-restore#3 0x7fb26c43e4d9 in __libc_start_main (/lib64/libc.so.6+0x204d9) checkpoint-restore#4 0x4031b9 in _start (/root/git/main/criu/test/zdtm/static/chroot+0x4031b9) Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>

==36==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60200000001c at pc 0x7fb26c88d5f9 bp 0x7ffc15087d40 sp 0x7ffc150874d0 WRITE of size 13 at 0x60200000001c thread T0 #0 0x7fb26c88d5f8 in vsprintf (/lib64/libasan.so.4+0x9e5f8) #1 0x7fb26c88d986 in __interceptor_sprintf (/lib64/libasan.so.4+0x9e986) #2 0x402453 in main /root/git/main/criu/test/zdtm/static/chroot.c:68 #3 0x7fb26c43e4d9 in __libc_start_main (/lib64/libc.so.6+0x204d9) #4 0x4031b9 in _start (/root/git/main/criu/test/zdtm/static/chroot+0x4031b9) Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>

CID 153098 (checkpoint-restore#3 of 3): Buffer not null terminated (BUFFER_SIZE_WARNING) 10. buffer_size_warning: Calling strncpy with a maximum size argument of 64 bytes on destination array m->fsname of size 64 bytes might leave the destination string unterminated. Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>

Segmentation fault was raised while trying to restore a process with tty. Coredump file says this is caused by uninitialized tty_mutex: (gdb) where #0 0x00000000004d7270 in atomic_add_return (i=1, v=0x0) at include/common/asm/atomic.h:34 #1 0x00000000004d7398 in mutex_lock (m=0x0) at include/common/lock.h:151 #2 0x00000000004d840c in __pty_open_ptmx_index (index=3, flags=2, cb=0x4dce50 <open_pty>, arg=0x11, path=0x5562e0 "ptmx") at criu/tty.c:603 #3 0x00000000004dced8 in pty_create_ptmx_index (dfd=17, index=3, flags=2) at criu/tty.c:2384 since init_tty_mutex() is reentrantable, just calling it before mutex_lock() Signed-off-by: Deng Guangxing <dengguangxing@huawei.com> Reviewed-by: Cyrill Gorcunov <gorcunov@gmail.com> Signed-off-by: Andrei Vagin <avagin@gmail.com>

CID 226485 (#1 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to CID 226485 (#2 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to CID 226485 (#3 of 3): Resource leak (RESOURCE_LEAK) Variable events going out of scope leaks the storage it points to Also changed epoll_prepare() to check return value of epoll_create() against '< 0' instead if '== -1' to make coverity happy. Signed-off-by: Adrian Reber <areber@redhat.com>

read_ns_sys_file() can return an error, but we are trying to parse a buffer before checking a return code. CID 417395 (#3 of 3): String not null terminated (STRING_NULL) 2. string_null: Passing unterminated string buf to strtol, which expects a null-terminated string. Signed-off-by: Andrei Vagin <avagin@gmail.com>

kimh changed the title ~~ˆRestoring Docker container with docker_cr.sh failed~~ Restoring Docker container with docker_cr.sh failed May 16, 2015

kimh closed this as completed May 19, 2015

kikosha mentioned this issue May 29, 2016

Segmentation fault in libnl-3.so when dumping a process #169

Closed

Qinghe12 mentioned this issue Oct 31, 2020

Error (criu/files.c:1210): Unable to open fd=9 id=0x4d #1251

Closed

This was referenced Mar 17, 2021

When name is null, it will coredump #1411

Closed

When name is null, it will coredump #1412

Open

throwbear mentioned this issue May 15, 2021

[Python Supervisord] criu restore is successful, But the supervisord program coredump #1477

Open

avagin mentioned this issue Oct 5, 2023

tun: don't parse buffers that have not been filled with data #2278

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restoring Docker container with docker_cr.sh failed #3

Restoring Docker container with docker_cr.sh failed #3

kimh commented May 16, 2015

xemul commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

tych0 commented May 18, 2015

avagin commented May 18, 2015 via email

boucher commented May 18, 2015

SaiedKazemi commented May 18, 2015

kimh commented May 19, 2015

Restoring Docker container with docker_cr.sh failed #3

Restoring Docker container with docker_cr.sh failed #3

Comments

kimh commented May 16, 2015

xemul commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

avagin commented May 18, 2015

kimh commented May 18, 2015

tych0 commented May 18, 2015

avagin commented May 18, 2015 via email

boucher commented May 18, 2015

SaiedKazemi commented May 18, 2015

kimh commented May 19, 2015