Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre-created cpuset cgroup with not configured .cpus/.mems causes restore to fail #16

Closed
klesgidisold opened this issue Aug 7, 2015 · 21 comments

Comments

@klesgidisold
Copy link

Hi! I 'm trying to restore a docker MySQL container to another host. So I checkpointed it, moved the dump files to another PC, created a new "identical" container (didn't run it) and executed

docker restore --force=true --image-dir /home/kyriakos/Desktop/c_r --work-dir ~/Desktop/logs container-name

The results

Error response from daemon: Cannot restore container mysql_impo: criu failed: type NOTIFY errno 0
Error: failed to restore one or more containers

The restore log

(00.000111) Added /sys/fs/cgroup/cpuset:/sys/fs/cgroup/cpuset ext mount mapping
(00.000151) Added /sys/fs/cgroup/cpu:/sys/fs/cgroup/cpu ext mount mapping
(00.000156) Added /sys/fs/cgroup/cpuacct:/sys/fs/cgroup/cpuacct ext mount mapping
(00.000159) Added /sys/fs/cgroup/memory:/sys/fs/cgroup/memory ext mount mapping
(00.000162) Added /sys/fs/cgroup/devices:/sys/fs/cgroup/devices ext mount mapping
(00.000165) Added /sys/fs/cgroup/freezer:/sys/fs/cgroup/freezer ext mount mapping
(00.000168) Added /sys/fs/cgroup/net_cls:/sys/fs/cgroup/net_cls ext mount mapping
(00.000171) Added /sys/fs/cgroup/blkio:/sys/fs/cgroup/blkio ext mount mapping
(00.000173) Added /sys/fs/cgroup/perf_event:/sys/fs/cgroup/perf_event ext mount mapping
(00.000176) Added /sys/fs/cgroup/net_prio:/sys/fs/cgroup/net_prio ext mount mapping
(00.000179) Added /sys/fs/cgroup/hugetlb:/sys/fs/cgroup/hugetlb ext mount mapping
(00.000182) Added /sys/fs/cgroup/systemd:/sys/fs/cgroup/systemd ext mount mapping
(00.000185) Added /var/lib/mysql:/var/lib/docker/volumes/da0a45864a7af547cbec0b3c1193fc1ba0b5f6a7da965820d011841c39ca3251/_data ext mount mapping
(00.000188) Added /etc/resolv.conf:/var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/resolv.conf ext mount mapping
(00.000191) Added /etc/hostname:/var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/hostname ext mount mapping
(00.000194) Added /etc/hosts:/var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/hosts ext mount mapping
(00.000277) TCP recv queue memory limit is 3145728
(00.000307) cpu: fpu:1 fxsr:1 xsave:1
(00.000405) vdso: Parsing at 7ffc65d79000 7ffc65d7b000
(00.000411) vdso: PT_LOAD p_vaddr: 0
(00.000414) vdso: DT_HASH: 0x120
(00.000417) vdso: DT_STRTAB: 0x268
(00.000420) vdso: DT_SYMTAB: 0x160
(00.000422) vdso: DT_STRSZ: 94
(00.000425) vdso: DT_SYMENT: 24
(00.000427) vdso: nbucket 3 nchain 11 bucket 0x7ffc65d79128 chain 0x7ffc65d79134
(00.000433) vdso: rt [vdso] 7ffc65d79000-7ffc65d7b000 [vvar] 7ffc65d77000-7ffc65d79000
(00.000465) Reading image tree
(00.000489) Add mnt ns 11 pid 1
(00.000502) Will restore in 6c020000 namespaces
(00.000506) NS mask to use 6c020000
(00.000512) Collecting 37/21 (flags 0)
(00.000527)  `- ... done
(00.000563) cg: Preparing cgroups yard (cgroups restore mode 0x4)
(00.001121) cg: Opening .criu.cgyard.xy4Eaz as cg yard
(00.001137) cg:     Making controller dir .criu.cgyard.xy4Eaz/cpuset (cpuset)
(00.001490) cg: Determined cgroup dir cpuset/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.001496) cg: Skip restoring properties on cgroup dir cpuset/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.001503) cg:     Making controller dir .criu.cgyard.xy4Eaz/cpu (cpu)
(00.001842) cg: Determined cgroup dir cpu/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.001848) cg: Skip restoring properties on cgroup dir cpu/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.001856) cg:     Making controller dir .criu.cgyard.xy4Eaz/cpuacct (cpuacct)
(00.002235) cg: Determined cgroup dir cpuacct/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.002279) cg: Skip restoring properties on cgroup dir cpuacct/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.002287) cg:     Making controller dir .criu.cgyard.xy4Eaz/memory (memory)
(00.003106) cg: Determined cgroup dir memory/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.003113) cg: Skip restoring properties on cgroup dir memory/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.003122) cg:     Making controller dir .criu.cgyard.xy4Eaz/devices (devices)
(00.003233) cg: Determined cgroup dir devices/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.003241) cg: Skip restoring properties on cgroup dir devices/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.003256) cg:     Making controller dir .criu.cgyard.xy4Eaz/freezer (freezer)
(00.003344) cg: Determined cgroup dir freezer/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.003349) cg: Skip restoring properties on cgroup dir freezer/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.003355) cg:     Making controller dir .criu.cgyard.xy4Eaz/net_cls (net_cls)
(00.003474) cg: Determined cgroup dir net_cls/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.003479) cg: Skip restoring properties on cgroup dir net_cls/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.003485) cg:     Making controller dir .criu.cgyard.xy4Eaz/blkio (blkio)
(00.003584) cg: Determined cgroup dir blkio/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.003588) cg: Skip restoring properties on cgroup dir blkio/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.003595) cg:     Making controller dir .criu.cgyard.xy4Eaz/perf_event (perf_event)
(00.003664) cg: Determined cgroup dir perf_event/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.003669) cg: Skip restoring properties on cgroup dir perf_event/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.003675) cg:     Making controller dir .criu.cgyard.xy4Eaz/net_prio (net_prio)
(00.003716) cg: Determined cgroup dir net_prio/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.003720) cg: Skip restoring properties on cgroup dir net_prio/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.003726) cg:     Making controller dir .criu.cgyard.xy4Eaz/hugetlb (hugetlb)
(00.003849) cg: Determined cgroup dir hugetlb/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573 already exist
(00.003854) cg: Skip restoring properties on cgroup dir hugetlb/docker/b8ea7fb3f4a0007a769804a101ba551356d6a3b1909ab242562310874fd9b573
(00.003862) cg:     Making controller dir .criu.cgyard.xy4Eaz/systemd (none,name=systemd)
(00.003958) cg: Determined cgroup dir systemd/ already exist
(00.003962) cg: Skip restoring properties on cgroup dir systemd/
(00.003968) cg: Determined cgroup dir systemd//user already exist
(00.003971) cg: Skip restoring properties on cgroup dir systemd//user
(00.003977) cg: Determined cgroup dir systemd//user/107.user already exist
(00.003980) cg: Skip restoring properties on cgroup dir systemd//user/107.user
(00.003986) cg: Determined cgroup dir systemd//user/107.user/c1.session already exist
(00.003990) cg: Skip restoring properties on cgroup dir systemd//user/107.user/c1.session
(00.003995) cg: Determined cgroup dir systemd//user/1000.user already exist
(00.003999) cg: Skip restoring properties on cgroup dir systemd//user/1000.user
(00.004005) cg: Determined cgroup dir systemd//user/1000.user/c2.session already exist
(00.004008) cg: Skip restoring properties on cgroup dir systemd//user/1000.user/c2.session
(00.004057) Warn  (cr-restore.c:1041): Set CLONE_PARENT | CLONE_NEWPID but it might cause restore problem,because not all kernels support such clone flags combinations!
(00.004062) Forking task with 1 pid (flags 0x6c028000)
(00.004077) Saved netns fd for links restore
(00.008215) PID: real 5148 virt 1
(00.008321) Wait until namespaces are created
(00.009932) Running setup-namespaces scripts
(00.009969)     RPC
(00.010122)      1: Restoring namespaces 1 flags 0x6c028000
(00.010604)      1: Restoring link lo type 1
(00.010879)      1: Restoring link eth0 type 2
(00.010884)      1: Restoring netdev eth0 idx 20
(00.010888)      1: Restore ll addr (02:../6) for device
(00.012052)      1: DEBUG Skip eth0/accept_local, val =0
(00.012060)      1: DEBUG Skip eth0/accept_redirects, val =1
(00.012063)      1: DEBUG Skip eth0/accept_source_route, val =1
(00.012066)      1: DEBUG Skip eth0/arp_accept, val =0
(00.012069)      1: DEBUG Skip eth0/arp_announce, val =0
(00.012072)      1: DEBUG Skip eth0/arp_filter, val =0
(00.012080)      1: DEBUG Skip eth0/arp_ignore, val =0
(00.012187)      1: DEBUG Skip eth0/arp_notify, val =0
(00.012192)      1: DEBUG Skip eth0/bootp_relay, val =0
(00.012195)      1: DEBUG Skip eth0/disable_policy, val =0
(00.012198)      1: DEBUG Skip eth0/disable_xfrm, val =0
(00.012200)      1: DEBUG Skip eth0/force_igmp_version, val =0
(00.012203)      1: DEBUG Skip eth0/forwarding, val =1
(00.012206)      1: DEBUG Skip eth0/igmpv2_unsolicited_report_interval, val =10000
(00.012209)      1: DEBUG Skip eth0/igmpv3_unsolicited_report_interval, val =1000
(00.012212)      1: DEBUG Skip eth0/log_martians, val =0
(00.012215)      1: DEBUG Skip eth0/medium_id, val =0
(00.012218)      1: DEBUG Skip eth0/promote_secondaries, val =0
(00.012290)      1: DEBUG Skip eth0/proxy_arp, val =0
(00.012296)      1: DEBUG Skip eth0/proxy_arp_pvlan, val =0
(00.012299)      1: DEBUG Skip eth0/route_localnet, val =0
(00.012301)      1: DEBUG Skip eth0/rp_filter, val =1
(00.012304)      1: DEBUG Skip eth0/secure_redirects, val =1
(00.012307)      1: DEBUG Skip eth0/send_redirects, val =1
(00.012310)      1: DEBUG Skip eth0/shared_media, val =1
(00.012313)      1: DEBUG Skip eth0/src_valid_mark, val =0
(00.012316)      1: DEBUG Skip eth0/tag, val =0
(00.012337)      1:     Running ip addr restore
(00.018219)      1: Found fd 1 (id pipe:[173136]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.018285)      1: Inherit fd 1 moved to 7 to resolve clash
RTNETLINK answers: File exists
RTNETLINK answers: File exists
(00.019724)      1:     Running ip route restore
(00.025395)      1: Found fd 1 (id pipe:[173136]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.025455)      1: Inherit fd 1 moved to 7 to resolve clash
RTNETLINK answers: File exists
(00.026933)      1:     Running iptables-restore for iptables-restore
(00.032704)      1: Found fd 1 (id pipe:[173136]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.032786)      1: Inherit fd 1 moved to 7 to resolve clash
(00.035404)      1: kernel/hostname nr 12
(00.035666)      1: kernel/domainname nr 6
(00.035683)      1: Restoring IPC namespace
(00.035687)      1: Restoring IPC variables
(00.035858)      1: Restoring IPC shared memory
(00.035865)      1: No ipcns-shm-9.img image
(00.035869)      1: Restoring IPC message queues
(00.035874)      1: No ipcns-msg-9.img image
(00.035877)      1: Restoring IPC semaphores sets
(00.035882)      1: No ipcns-sem-9.img image
(00.035886)      1: Restoring mount namespace
(00.036011)      1:     type ext4 source /dev/disk/by-uuid/b2c933d5-6fa6-4035-980a-ac7ae5fe64d2 mnt_id 65 s_dev 0x800006 / @ ./ flags 0x200000 options errors=remount-ro,data=ordered
(00.036100)      1:     type devtmpfs source udev mnt_id 66 s_dev 0x6 / @ ./dev flags 0x200000 options size=1663008k,nr_inodes=415752,mode=755
(00.036110)      1:     type devpts source devpts mnt_id 67 s_dev 0xd / @ ./dev/pts flags 0x20000a options gid=5,mode=620,ptmxmode=000
(00.036159)      1:     type tmpfs source tmpfs mnt_id 68 s_dev 0x11 / @ ./run flags 0x20000a options size=333076k,mode=755
(00.036170)      1:     type tmpfs source none mnt_id 69 s_dev 0x14 / @ ./run/lock flags 0x20000e options size=5120k
(00.036178)      1:     type tmpfs source none mnt_id 70 s_dev 0x15 / @ ./run/shm flags 0x200006 options 
(00.036187)      1:     type tmpfs source none mnt_id 71 s_dev 0x16 / @ ./run/user flags 0x20000e options size=102400k,mode=755
(00.036195)      1:     type fuse source gvfsd-fuse mnt_id 72 s_dev 0x26 / @ ./run/user/1000/gvfs flags 0x200006 options user_id=1000,group_id=1000
(00.036204)      1:     type fuse source vmware-vmblock mnt_id 73 s_dev 0x25 / @ ./run/vmblock-fuse flags 0x200006 options user_id=0,group_id=0,default_permissions,allow_other
(00.036212)      1:     type nsfs source nsfs mnt_id 74 s_dev 0x3 / @ ./run/docker/netns/16e94965a2a4 flags 0x1000000 options 
(00.036327)      1:     type aufs source none mnt_id 75 s_dev 0x27 / @ ./run/docker/execdriver/native/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/criu-root flags 0x200000 options si=f7520c4da969c7db,dio,dirperm1
(00.036352)      1:     type sysfs source sysfs mnt_id 76 s_dev 0x10 / @ ./sys flags 0x20000e options 
(00.036389)      1:     type tmpfs source none mnt_id 77 s_dev 0x12 / @ ./sys/fs/cgroup flags 0x200000 options size=4k,mode=755
(00.036398)      1:     type cgroup source cgroup mnt_id 78 s_dev 0x18 / @ ./sys/fs/cgroup/cpuset flags 0x200000 options cpuset
(00.036406)      1:     type cgroup source cgroup mnt_id 79 s_dev 0x19 / @ ./sys/fs/cgroup/cpu flags 0x200000 options cpu
(00.036414)      1:     type cgroup source cgroup mnt_id 80 s_dev 0x1a / @ ./sys/fs/cgroup/cpuacct flags 0x200000 options cpuacct
(00.036422)      1:     type cgroup source cgroup mnt_id 81 s_dev 0x1b / @ ./sys/fs/cgroup/memory flags 0x200000 options memory
(00.036430)      1:     type cgroup source cgroup mnt_id 82 s_dev 0x1c / @ ./sys/fs/cgroup/devices flags 0x200000 options devices
(00.036437)      1:     type cgroup source cgroup mnt_id 83 s_dev 0x1d / @ ./sys/fs/cgroup/freezer flags 0x200000 options freezer
(00.036447)      1:     type cgroup source cgroup mnt_id 84 s_dev 0x1e / @ ./sys/fs/cgroup/net_cls flags 0x200000 options net_cls
(00.036456)      1:     type cgroup source cgroup mnt_id 85 s_dev 0x1f / @ ./sys/fs/cgroup/blkio flags 0x200000 options blkio
(00.036463)      1:     type cgroup source cgroup mnt_id 86 s_dev 0x20 / @ ./sys/fs/cgroup/perf_event flags 0x200000 options perf_event
(00.036471)      1:     type cgroup source cgroup mnt_id 87 s_dev 0x21 / @ ./sys/fs/cgroup/net_prio flags 0x200000 options net_prio
(00.036506)      1:     type cgroup source cgroup mnt_id 88 s_dev 0x22 / @ ./sys/fs/cgroup/hugetlb flags 0x200000 options hugetlb
(00.036562)      1:     type cgroup source systemd mnt_id 89 s_dev 0x23 / @ ./sys/fs/cgroup/systemd flags 0x20000e options name=systemd
(00.036572)      1:     type fusectl source none mnt_id 90 s_dev 0x13 / @ ./sys/fs/fuse/connections flags 0x200000 options 
(00.036580)      1:     type debugfs source none mnt_id 91 s_dev 0x7 / @ ./sys/kernel/debug flags 0x200000 options 
(00.036662)      1:     type securityfs source none mnt_id 92 s_dev 0xb / @ ./sys/kernel/security flags 0x200000 options 
(00.036671)      1:     type pstore source none mnt_id 93 s_dev 0x17 / @ ./sys/fs/pstore flags 0x200000 options 
(00.036679)      1:     type proc source proc mnt_id 94 s_dev 0x4 / @ ./proc flags 0x20000e options 
(00.036717)      1:     type vfat source /dev/sdf1 mnt_id 95 s_dev 0x800051 / @ ./media/kyriakos/UBUNTU 14_0 flags 0x200006 options uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,showexec,utf8,flush,errors=remount-ro
(00.036729)      1:     type ext4 source /dev/disk/by-uuid/b2c933d5-6fa6-4035-980a-ac7ae5fe64d2 mnt_id 96 s_dev 0x800006 /var/lib/docker/aufs @ ./var/lib/docker/aufs flags 0x200000 options errors=remount-ro,data=ordered
(00.036738)      1:     type aufs source none mnt_id 97 s_dev 0x27 / @ ./var/lib/docker/aufs/mnt/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743 flags 0x200000 options si=f7520c4da969c7db,dio,dirperm1
(00.036765)      1:     type tmpfs source none mnt_id 98 s_dev 0x28 / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz flags 0x200000 options 
(00.036774)      1:     type cgroup source none mnt_id 99 s_dev 0x18 / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/cpuset flags 0x200000 options cpuset
(00.036785)      1:     type cgroup source none mnt_id 100 s_dev 0x19 / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/cpu flags 0x200000 options cpu
(00.036793)      1:     type cgroup source none mnt_id 101 s_dev 0x1a / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/cpuacct flags 0x200000 options cpuacct
(00.036801)      1:     type cgroup source none mnt_id 102 s_dev 0x1b / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/memory flags 0x200000 options memory
(00.036881)      1:     type cgroup source none mnt_id 103 s_dev 0x1c / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/devices flags 0x200000 options devices
(00.036892)      1:     type cgroup source none mnt_id 104 s_dev 0x1d / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/freezer flags 0x200000 options freezer
(00.036905)      1:     type cgroup source none mnt_id 105 s_dev 0x1e / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/net_cls flags 0x200000 options net_cls
(00.036941)      1:     type cgroup source none mnt_id 106 s_dev 0x1f / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/blkio flags 0x200000 options blkio
(00.036953)      1:     type cgroup source none mnt_id 107 s_dev 0x20 / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/perf_event flags 0x200000 options perf_event
(00.036968)      1:     type cgroup source none mnt_id 108 s_dev 0x21 / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/net_prio flags 0x200000 options net_prio
(00.036976)      1:     type cgroup source none mnt_id 109 s_dev 0x22 / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/hugetlb flags 0x200000 options hugetlb
(00.036985)      1:     type cgroup source none mnt_id 110 s_dev 0x23 / @ ./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/systemd flags 0x200000 options name=systemd
(00.036997)      1: Building mountpoints tree
(00.037001)      1:     Building plain mount tree
(00.037004)      1:         Working on 110->98
(00.037007)      1:         Working on 109->98
(00.037010)      1:         Working on 108->98
(00.037012)      1:         Working on 107->98
(00.037015)      1:         Working on 106->98
(00.037018)      1:         Working on 105->98
(00.037021)      1:         Working on 104->98
(00.037024)      1:         Working on 103->98
(00.037026)      1:         Working on 102->98
(00.037100)      1:         Working on 101->98
(00.037105)      1:         Working on 100->98
(00.037108)      1:         Working on 99->98
(00.037111)      1:         Working on 98->65
(00.037114)      1:         Working on 97->96
(00.037117)      1:         Working on 96->65
(00.037120)      1:         Working on 95->65
(00.037122)      1:         Working on 94->65
(00.037125)      1:         Working on 93->76
(00.037128)      1:         Working on 92->76
(00.037131)      1:         Working on 91->76
(00.037161)      1:         Working on 90->76
(00.037165)      1:         Working on 89->77
(00.037168)      1:         Working on 88->77
(00.037171)      1:         Working on 87->77
(00.037174)      1:         Working on 86->77
(00.037176)      1:         Working on 85->77
(00.037179)      1:         Working on 84->77
(00.037182)      1:         Working on 83->77
(00.037185)      1:         Working on 82->77
(00.037187)      1:         Working on 81->77
(00.037190)      1:         Working on 80->77
(00.037193)      1:         Working on 79->77
(00.037195)      1:         Working on 78->77
(00.037198)      1:         Working on 77->76
(00.037201)      1:         Working on 76->65
(00.037204)      1:         Working on 75->68
(00.037207)      1:         Working on 74->68
(00.037210)      1:         Working on 73->68
(00.037212)      1:         Working on 72->71
(00.037215)      1:         Working on 71->68
(00.037218)      1:         Working on 70->68
(00.037221)      1:         Working on 69->68
(00.037223)      1:         Working on 68->65
(00.037226)      1:         Working on 67->66
(00.037229)      1:         Working on 66->65
(00.037232)      1:         Working on 65->64
(00.037235)      1:     Resorting siblings on 65
(00.037237)      1:     Resorting siblings on 98
(00.037240)      1:     Resorting siblings on 110
(00.037243)      1:     Resorting siblings on 109
(00.037246)      1:     Resorting siblings on 108
(00.037318)      1:     Resorting siblings on 107
(00.037323)      1:     Resorting siblings on 106
(00.037326)      1:     Resorting siblings on 105
(00.037329)      1:     Resorting siblings on 104
(00.037332)      1:     Resorting siblings on 103
(00.037335)      1:     Resorting siblings on 102
(00.037338)      1:     Resorting siblings on 101
(00.037340)      1:     Resorting siblings on 100
(00.037343)      1:     Resorting siblings on 99
(00.037346)      1:     Resorting siblings on 96
(00.037349)      1:     Resorting siblings on 97
(00.037378)      1:     Resorting siblings on 95
(00.037383)      1:     Resorting siblings on 94
(00.037386)      1:     Resorting siblings on 76
(00.037389)      1:     Resorting siblings on 93
(00.037392)      1:     Resorting siblings on 92
(00.037394)      1:     Resorting siblings on 91
(00.037397)      1:     Resorting siblings on 90
(00.037400)      1:     Resorting siblings on 77
(00.037403)      1:     Resorting siblings on 89
(00.037406)      1:     Resorting siblings on 88
(00.037413)      1:     Resorting siblings on 87
(00.037416)      1:     Resorting siblings on 86
(00.037419)      1:     Resorting siblings on 85
(00.037422)      1:     Resorting siblings on 84
(00.037425)      1:     Resorting siblings on 83
(00.037427)      1:     Resorting siblings on 82
(00.037430)      1:     Resorting siblings on 81
(00.037433)      1:     Resorting siblings on 80
(00.037436)      1:     Resorting siblings on 79
(00.037438)      1:     Resorting siblings on 78
(00.037441)      1:     Resorting siblings on 68
(00.037444)      1:     Resorting siblings on 75
(00.037447)      1:     Resorting siblings on 74
(00.037450)      1:     Resorting siblings on 73
(00.037452)      1:     Resorting siblings on 71
(00.037455)      1:     Resorting siblings on 72
(00.037458)      1:     Resorting siblings on 70
(00.037460)      1:     Resorting siblings on 69
(00.037463)      1:     Resorting siblings on 66
(00.037466)      1:     Resorting siblings on 67
(00.037469)      1: Done:
(00.037471)      1: [./](65->64)
(00.037475)      1:  [./proc](94->65)
(00.037478)      1:  <--
(00.037480)      1:  [./dev](66->65)
(00.037483)      1:   [./dev/pts](67->66)
(00.037486)      1:   <--
(00.037489)      1:  <--
(00.037492)      1:  [./run](68->65)
(00.037495)      1:   [./run/vmblock-fuse](73->68)
(00.037498)      1:   <--
(00.037500)      1:   [./run/lock](69->68)
(00.037503)      1:   <--
(00.037506)      1:   [./run/shm](70->68)
(00.037509)      1:   <--
(00.037511)      1:   [./run/user](71->68)
(00.037514)      1:    [./run/user/1000/gvfs](72->71)
(00.037517)      1:    <--
(00.037520)      1:   <--
(00.037522)      1:   [./run/docker/netns/16e94965a2a4](74->68)
(00.037525)      1:   <--
(00.037528)      1:   [./run/docker/execdriver/native/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/criu-root](75->68)
(00.037531)      1:   <--
(00.037534)      1:  <--
(00.037537)      1:  [./sys](76->65)
(00.037540)      1:   [./sys/fs/pstore](93->76)
(00.037543)      1:   <--
(00.037545)      1:   [./sys/fs/cgroup](77->76)
(00.037548)      1:    [./sys/fs/cgroup/systemd](89->77)
(00.037551)      1:    <--
(00.037554)      1:    [./sys/fs/cgroup/cpuset](78->77)
(00.037557)      1:    <--
(00.037559)      1:    [./sys/fs/cgroup/cpu](79->77)
(00.037562)      1:    <--
(00.037565)      1:    [./sys/fs/cgroup/cpuacct](80->77)
(00.037568)      1:    <--
(00.037571)      1:    [./sys/fs/cgroup/memory](81->77)
(00.037574)      1:    <--
(00.037576)      1:    [./sys/fs/cgroup/devices](82->77)
(00.037579)      1:    <--
(00.037582)      1:    [./sys/fs/cgroup/freezer](83->77)
(00.037585)      1:    <--
(00.037587)      1:    [./sys/fs/cgroup/net_cls](84->77)
(00.037590)      1:    <--
(00.037593)      1:    [./sys/fs/cgroup/blkio](85->77)
(00.037596)      1:    <--
(00.037599)      1:    [./sys/fs/cgroup/perf_event](86->77)
(00.037602)      1:    <--
(00.037604)      1:    [./sys/fs/cgroup/net_prio](87->77)
(00.037607)      1:    <--
(00.037610)      1:    [./sys/fs/cgroup/hugetlb](88->77)
(00.037613)      1:    <--
(00.037615)      1:   <--
(00.037618)      1:   [./sys/fs/fuse/connections](90->76)
(00.037621)      1:   <--
(00.037624)      1:   [./sys/kernel/debug](91->76)
(00.037627)      1:   <--
(00.037629)      1:   [./sys/kernel/security](92->76)
(00.037632)      1:   <--
(00.037635)      1:  <--
(00.037638)      1:  [./media/kyriakos/UBUNTU 14_0](95->65)
(00.037641)      1:  <--
(00.037643)      1:  [./var/lib/docker/aufs](96->65)
(00.037646)      1:   [./var/lib/docker/aufs/mnt/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743](97->96)
(00.037649)      1:   <--
(00.037652)      1:  <--
(00.037655)      1:  [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz](98->65)
(00.037658)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/systemd](110->98)
(00.037661)      1:   <--
(00.037664)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/cpuset](99->98)
(00.037667)      1:   <--
(00.037669)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/cpu](100->98)
(00.037672)      1:   <--
(00.037675)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/cpuacct](101->98)
(00.037681)      1:   <--
(00.037685)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/memory](102->98)
(00.037688)      1:   <--
(00.037690)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/devices](103->98)
(00.037694)      1:   <--
(00.037696)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/freezer](104->98)
(00.037699)      1:   <--
(00.037702)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/net_cls](105->98)
(00.037705)      1:   <--
(00.037708)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/blkio](106->98)
(00.037711)      1:   <--
(00.037713)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/perf_event](107->98)
(00.037716)      1:   <--
(00.037719)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/net_prio](108->98)
(00.037722)      1:   <--
(00.037725)      1:   [./home/kyriakos/Desktop/logs/.criu.cgyard.xy4Eaz/hugetlb](109->98)
(00.037728)      1:   <--
(00.037731)      1:  <--
(00.037733)      1: <--
(00.037749)      1: Reading mountpoint images
(00.037756)      1:         Getting source for 60
(00.037759)      1:         Getting root for 60
(00.037763)      1:         Getting mpt for 60 ./proc/timer_stats
(00.037766)      1:         Getting opts for 60
(00.037769)      1:     Read 60 mp @ ./proc/timer_stats
(00.037773)      1:         Getting source for 59
(00.037777)      1:         Getting root for 59
(00.037780)      1:         Getting mpt for 59 ./proc/kcore
(00.037783)      1:         Getting opts for 59
(00.037786)      1:     Read 59 mp @ ./proc/kcore
(00.037790)      1:         Getting source for 58
(00.037793)      1:         Getting root for 58
(00.037797)      1:         Getting mpt for 58 ./proc/sysrq-trigger
(00.037800)      1:         Getting opts for 58
(00.037802)      1:     Read 58 mp @ ./proc/sysrq-trigger
(00.037806)      1:         Getting source for 57
(00.037810)      1:         Getting root for 57
(00.037813)      1:         Getting mpt for 57 ./proc/sys
(00.037816)      1:         Getting opts for 57
(00.037819)      1:     Read 57 mp @ ./proc/sys
(00.037823)      1:         Getting source for 56
(00.037826)      1:         Getting root for 56
(00.037829)      1:         Getting mpt for 56 ./proc/irq
(00.037832)      1:         Getting opts for 56
(00.037835)      1:     Read 56 mp @ ./proc/irq
(00.037839)      1:         Getting source for 55
(00.037842)      1:         Getting root for 55
(00.037845)      1:         Getting mpt for 55 ./proc/fs
(00.037848)      1:         Getting opts for 55
(00.037851)      1:     Read 55 mp @ ./proc/fs
(00.037855)      1:         Getting source for 54
(00.037859)      1:         Getting root for 54
(00.037862)      1:         Getting mpt for 54 ./proc/bus
(00.037865)      1:         Getting opts for 54
(00.037868)      1:     Read 54 mp @ ./proc/bus
(00.037872)      1:         Getting source for 53
(00.037875)      1:         Getting root for 53
(00.037878)      1:         Getting mpt for 53 ./proc/asound
(00.037881)      1:         Getting opts for 53
(00.037884)      1:     Read 53 mp @ ./proc/asound
(00.037888)      1:         Getting source for 111
(00.037893)      1: Mountpoint /etc/hosts will have root from /var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/hosts
(00.037897)      1:         Getting mpt for 111 ./etc/hosts
(00.037900)      1:         Getting opts for 111
(00.037902)      1:     Read 111 mp @ ./etc/hosts
(00.037907)      1:         Getting source for 110
(00.037910)      1: Mountpoint /etc/hostname will have root from /var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/hostname
(00.037914)      1:         Getting mpt for 110 ./etc/hostname
(00.037917)      1:         Getting opts for 110
(00.037919)      1:     Read 110 mp @ ./etc/hostname
(00.037924)      1:         Getting source for 109
(00.037927)      1: Mountpoint /etc/resolv.conf will have root from /var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/resolv.conf
(00.037931)      1:         Getting mpt for 109 ./etc/resolv.conf
(00.037933)      1:         Getting opts for 109
(00.037936)      1:     Read 109 mp @ ./etc/resolv.conf
(00.037940)      1:         Getting source for 108
(00.037944)      1: Mountpoint /var/lib/mysql will have root from /var/lib/docker/volumes/da0a45864a7af547cbec0b3c1193fc1ba0b5f6a7da965820d011841c39ca3251/_data
(00.037952)      1:         Getting mpt for 108 ./var/lib/mysql
(00.037955)      1:         Getting opts for 108
(00.037958)      1:     Read 108 mp @ ./var/lib/mysql
(00.037962)      1:         Getting source for 107
(00.037966)      1: Mountpoint /sys/fs/cgroup/systemd will have root from /sys/fs/cgroup/systemd
(00.037969)      1:         Getting mpt for 107 ./sys/fs/cgroup/systemd
(00.037972)      1:         Getting opts for 107
(00.037975)      1:     Read 107 mp @ ./sys/fs/cgroup/systemd
(00.037979)      1:         Getting source for 106
(00.037983)      1: Mountpoint /sys/fs/cgroup/hugetlb will have root from /sys/fs/cgroup/hugetlb
(00.037986)      1:         Getting mpt for 106 ./sys/fs/cgroup/hugetlb
(00.037989)      1:         Getting opts for 106
(00.037992)      1:     Read 106 mp @ ./sys/fs/cgroup/hugetlb
(00.037998)      1:         Getting source for 105
(00.038002)      1: Mountpoint /sys/fs/cgroup/net_prio will have root from /sys/fs/cgroup/net_prio
(00.038005)      1:         Getting mpt for 105 ./sys/fs/cgroup/net_prio
(00.038008)      1:         Getting opts for 105
(00.038011)      1:     Read 105 mp @ ./sys/fs/cgroup/net_prio
(00.038015)      1:         Getting source for 104
(00.038019)      1: Mountpoint /sys/fs/cgroup/perf_event will have root from /sys/fs/cgroup/perf_event
(00.038022)      1:         Getting mpt for 104 ./sys/fs/cgroup/perf_event
(00.038025)      1:         Getting opts for 104
(00.038028)      1:     Read 104 mp @ ./sys/fs/cgroup/perf_event
(00.038032)      1:         Getting source for 103
(00.038035)      1: Mountpoint /sys/fs/cgroup/blkio will have root from /sys/fs/cgroup/blkio
(00.038039)      1:         Getting mpt for 103 ./sys/fs/cgroup/blkio
(00.038042)      1:         Getting opts for 103
(00.038044)      1:     Read 103 mp @ ./sys/fs/cgroup/blkio
(00.038048)      1:         Getting source for 102
(00.038052)      1: Mountpoint /sys/fs/cgroup/net_cls will have root from /sys/fs/cgroup/net_cls
(00.038055)      1:         Getting mpt for 102 ./sys/fs/cgroup/net_cls
(00.038058)      1:         Getting opts for 102
(00.038061)      1:     Read 102 mp @ ./sys/fs/cgroup/net_cls
(00.038065)      1:         Getting source for 101
(00.038068)      1: Mountpoint /sys/fs/cgroup/freezer will have root from /sys/fs/cgroup/freezer
(00.038071)      1:         Getting mpt for 101 ./sys/fs/cgroup/freezer
(00.038074)      1:         Getting opts for 101
(00.038077)      1:     Read 101 mp @ ./sys/fs/cgroup/freezer
(00.038081)      1:         Getting source for 100
(00.038085)      1: Mountpoint /sys/fs/cgroup/devices will have root from /sys/fs/cgroup/devices
(00.038088)      1:         Getting mpt for 100 ./sys/fs/cgroup/devices
(00.038091)      1:         Getting opts for 100
(00.038094)      1:     Read 100 mp @ ./sys/fs/cgroup/devices
(00.038100)      1:         Getting source for 99
(00.038103)      1: Mountpoint /sys/fs/cgroup/memory will have root from /sys/fs/cgroup/memory
(00.038107)      1:         Getting mpt for 99 ./sys/fs/cgroup/memory
(00.038110)      1:         Getting opts for 99
(00.038112)      1:     Read 99 mp @ ./sys/fs/cgroup/memory
(00.038117)      1:         Getting source for 98
(00.038120)      1: Mountpoint /sys/fs/cgroup/cpuacct will have root from /sys/fs/cgroup/cpuacct
(00.038123)      1:         Getting mpt for 98 ./sys/fs/cgroup/cpuacct
(00.038126)      1:         Getting opts for 98
(00.038129)      1:     Read 98 mp @ ./sys/fs/cgroup/cpuacct
(00.038133)      1:         Getting source for 97
(00.038136)      1: Mountpoint /sys/fs/cgroup/cpu will have root from /sys/fs/cgroup/cpu
(00.038139)      1:         Getting mpt for 97 ./sys/fs/cgroup/cpu
(00.038142)      1:         Getting opts for 97
(00.038145)      1:     Read 97 mp @ ./sys/fs/cgroup/cpu
(00.038149)      1:         Getting source for 96
(00.038152)      1: Mountpoint /sys/fs/cgroup/cpuset will have root from /sys/fs/cgroup/cpuset
(00.038163)      1:         Getting mpt for 96 ./sys/fs/cgroup/cpuset
(00.038167)      1:         Getting opts for 96
(00.038170)      1:     Read 96 mp @ ./sys/fs/cgroup/cpuset
(00.038174)      1:         Getting source for 95
(00.038178)      1:         Getting root for 95
(00.038181)      1:         Getting mpt for 95 ./sys/fs/cgroup
(00.038251)      1:         Getting opts for 95
(00.038255)      1:     Read 95 mp @ ./sys/fs/cgroup
(00.038260)      1:         Getting source for 94
(00.038263)      1:         Getting root for 94
(00.038266)      1:         Getting mpt for 94 ./sys
(00.038269)      1:         Getting opts for 94
(00.038272)      1:     Read 94 mp @ ./sys
(00.038276)      1:         Getting source for 93
(00.038279)      1:         Getting root for 93
(00.038284)      1:         Getting mpt for 93 ./dev/mqueue
(00.038288)      1:         Getting opts for 93
(00.038291)      1:     Read 93 mp @ ./dev/mqueue
(00.038295)      1:         Getting source for 92
(00.038298)      1:         Getting root for 92
(00.038301)      1:         Getting mpt for 92 ./dev/shm
(00.038304)      1:         Getting opts for 92
(00.038307)      1:     Read 92 mp @ ./dev/shm
(00.038311)      1:         Getting source for 91
(00.038314)      1:         Getting root for 91
(00.038317)      1:         Getting mpt for 91 ./dev/pts
(00.038320)      1:         Getting opts for 91
(00.038322)      1:     Read 91 mp @ ./dev/pts
(00.038326)      1:         Getting source for 90
(00.038330)      1:         Getting root for 90
(00.038333)      1:         Getting mpt for 90 ./dev
(00.038336)      1:         Getting opts for 90
(00.038338)      1:     Read 90 mp @ ./dev
(00.038342)      1:         Getting source for 89
(00.038346)      1:         Getting root for 89
(00.038349)      1:         Getting mpt for 89 ./proc
(00.038352)      1:         Getting opts for 89
(00.038354)      1:     Read 89 mp @ ./proc
(00.038358)      1:         Getting source for 88
(00.038362)      1:         Getting root for 88
(00.038365)      1:         Getting mpt for 88 ./
(00.038367)      1:         Getting opts for 88
(00.038370)      1:     Read 88 mp @ ./
(00.038384)      1: Path `/run/docker/execdriver/native/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/criu-root' resolved to `./run/docker/execdriver/native/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/criu-root' mountpoint
(00.038410)      1: Building mountpoints tree
(00.038414)      1:     Building plain mount tree
(00.038416)      1:         Working on 88->52
(00.038419)      1:         Working on 89->88
(00.038422)      1:         Working on 90->88
(00.038425)      1:         Working on 91->90
(00.038427)      1:         Working on 92->90
(00.038430)      1:         Working on 93->90
(00.038433)      1:         Working on 94->88
(00.038436)      1:         Working on 95->94
(00.038438)      1:         Working on 96->95
(00.038441)      1:         Working on 97->95
(00.038444)      1:         Working on 98->95
(00.038446)      1:         Working on 99->95
(00.038449)      1:         Working on 100->95
(00.038452)      1:         Working on 101->95
(00.038454)      1:         Working on 102->95
(00.038457)      1:         Working on 103->95
(00.038460)      1:         Working on 104->95
(00.038462)      1:         Working on 105->95
(00.038465)      1:         Working on 106->95
(00.038468)      1:         Working on 107->95
(00.038470)      1:         Working on 108->88
(00.038473)      1:         Working on 109->88
(00.038476)      1:         Working on 110->88
(00.038478)      1:         Working on 111->88
(00.038481)      1:         Working on 53->89
(00.038484)      1:         Working on 54->89
(00.038486)      1:         Working on 55->89
(00.038489)      1:         Working on 56->89
(00.038491)      1:         Working on 57->89
(00.038494)      1:         Working on 58->89
(00.038497)      1:         Working on 59->89
(00.038499)      1:         Working on 60->89
(00.038502)      1:     Resorting siblings on 88
(00.038505)      1:     Resorting siblings on 89
(00.038507)      1:     Resorting siblings on 53
(00.038510)      1:     Resorting siblings on 54
(00.038513)      1:     Resorting siblings on 55
(00.038515)      1:     Resorting siblings on 56
(00.038518)      1:     Resorting siblings on 57
(00.038521)      1:     Resorting siblings on 58
(00.038523)      1:     Resorting siblings on 59
(00.038526)      1:     Resorting siblings on 60
(00.038529)      1:     Resorting siblings on 90
(00.038531)      1:     Resorting siblings on 91
(00.038534)      1:     Resorting siblings on 92
(00.038537)      1:     Resorting siblings on 93
(00.038539)      1:     Resorting siblings on 94
(00.038542)      1:     Resorting siblings on 95
(00.038545)      1:     Resorting siblings on 96
(00.038547)      1:     Resorting siblings on 97
(00.038554)      1:     Resorting siblings on 98
(00.038557)      1:     Resorting siblings on 99
(00.038560)      1:     Resorting siblings on 100
(00.038563)      1:     Resorting siblings on 101
(00.038566)      1:     Resorting siblings on 102
(00.038568)      1:     Resorting siblings on 103
(00.038571)      1:     Resorting siblings on 104
(00.038574)      1:     Resorting siblings on 105
(00.038576)      1:     Resorting siblings on 106
(00.038579)      1:     Resorting siblings on 107
(00.038582)      1:     Resorting siblings on 108
(00.038585)      1:     Resorting siblings on 109
(00.038587)      1:     Resorting siblings on 110
(00.038590)      1:     Resorting siblings on 111
(00.038593)      1: Done:
(00.038595)      1: [./](88->52)
(00.038598)      1:  [./proc](89->88)
(00.038601)      1:   [./proc/asound](53->89)
(00.038604)      1:   <--
(00.038607)      1:   [./proc/timer_stats](60->89)
(00.038610)      1:   <--
(00.038612)      1:   [./proc/kcore](59->89)
(00.038615)      1:   <--
(00.038618)      1:   [./proc/sysrq-trigger](58->89)
(00.038621)      1:   <--
(00.038623)      1:   [./proc/sys](57->89)
(00.038626)      1:   <--
(00.038629)      1:   [./proc/irq](56->89)
(00.038632)      1:   <--
(00.038634)      1:   [./proc/fs](55->89)
(00.038637)      1:   <--
(00.038640)      1:   [./proc/bus](54->89)
(00.038643)      1:   <--
(00.038646)      1:  <--
(00.038648)      1:  [./etc/hosts](111->88)
(00.038651)      1:  <--
(00.038654)      1:  [./etc/hostname](110->88)
(00.038657)      1:  <--
(00.038659)      1:  [./etc/resolv.conf](109->88)
(00.038662)      1:  <--
(00.038665)      1:  [./var/lib/mysql](108->88)
(00.038668)      1:  <--
(00.038671)      1:  [./sys](94->88)
(00.038673)      1:   [./sys/fs/cgroup](95->94)
(00.038676)      1:    [./sys/fs/cgroup/cpuset](96->95)
(00.038679)      1:    <--
(00.038682)      1:    [./sys/fs/cgroup/systemd](107->95)
(00.038685)      1:    <--
(00.038688)      1:    [./sys/fs/cgroup/hugetlb](106->95)
(00.038691)      1:    <--
(00.038693)      1:    [./sys/fs/cgroup/net_prio](105->95)
(00.038696)      1:    <--
(00.038699)      1:    [./sys/fs/cgroup/perf_event](104->95)
(00.038702)      1:    <--
(00.038705)      1:    [./sys/fs/cgroup/blkio](103->95)
(00.038708)      1:    <--
(00.038711)      1:    [./sys/fs/cgroup/net_cls](102->95)
(00.038714)      1:    <--
(00.038716)      1:    [./sys/fs/cgroup/freezer](101->95)
(00.038719)      1:    <--
(00.038722)      1:    [./sys/fs/cgroup/devices](100->95)
(00.038725)      1:    <--
(00.038728)      1:    [./sys/fs/cgroup/memory](99->95)
(00.038731)      1:    <--
(00.038733)      1:    [./sys/fs/cgroup/cpuacct](98->95)
(00.038736)      1:    <--
(00.038739)      1:    [./sys/fs/cgroup/cpu](97->95)
(00.038742)      1:    <--
(00.038745)      1:   <--
(00.038747)      1:  <--
(00.038750)      1:  [./dev](90->88)
(00.038753)      1:   [./dev/pts](91->90)
(00.038756)      1:   <--
(00.038758)      1:   [./dev/mqueue](93->90)
(00.038761)      1:   <--
(00.038764)      1:   [./dev/shm](92->90)
(00.038767)      1:   <--
(00.038770)      1:  <--
(00.038772)      1: <--
(00.038783)      1: 88:./ private 1 shared 0 slave 0
(00.038790)      1: Start with 88:./
(00.038793)      1:     Mounting aufs @./ (0)
(00.038797)      1:     Mounting proc @./proc (0)
(00.038820)      1: 89:./proc private 1 shared 0 slave 0
(00.038827)      1:     Mounting proc @./proc/asound (0)
(00.038831)      1:     Bind ./proc/asound to ./proc/asound
(00.038844)      1: 53:./proc/asound private 1 shared 0 slave 0
(00.038851)      1: Postpone slave ./proc/timer_stats
(00.038855)      1: Postpone slave ./proc/kcore
(00.038858)      1:     Mounting proc @./proc/sysrq-trigger (0)
(00.038861)      1:     Bind ./proc/sysrq-trigger to ./proc/sysrq-trigger
(00.038872)      1: 58:./proc/sysrq-trigger private 1 shared 0 slave 0
(00.038879)      1:     Mounting proc @./proc/sys (0)
(00.038883)      1:     Bind ./proc/sys to ./proc/sys
(00.038892)      1: 57:./proc/sys private 1 shared 0 slave 0
(00.038899)      1:     Mounting proc @./proc/irq (0)
(00.038903)      1:     Bind ./proc/irq to ./proc/irq
(00.038916)      1: 56:./proc/irq private 1 shared 0 slave 0
(00.038923)      1:     Mounting proc @./proc/fs (0)
(00.038928)      1:     Bind ./proc/fs to ./proc/fs
(00.038936)      1: 55:./proc/fs private 1 shared 0 slave 0
(00.038943)      1:     Mounting proc @./proc/bus (0)
(00.038947)      1:     Bind ./proc/bus to ./proc/bus
(00.038956)      1: 54:./proc/bus private 1 shared 0 slave 0
(00.038962)      1:     Mounting unsupported @./etc/hosts (0)
(00.038966)      1:     Bind /var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/hosts to ./etc/hosts
(00.038981)      1: 111:./etc/hosts private 1 shared 0 slave 0
(00.038992)      1:     Mounting unsupported @./etc/hostname (0)
(00.038996)      1:     Bind /var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/hostname to ./etc/hostname
(00.039006)      1: 110:./etc/hostname private 1 shared 0 slave 0
(00.039016)      1:     Mounting unsupported @./etc/resolv.conf (0)
(00.039020)      1:     Bind /var/lib/docker/containers/16e94965a2a49562b10aa74c98e5e53ec0591bfb1279743455af5e6646422743/resolv.conf to ./etc/resolv.conf
(00.039029)      1: 109:./etc/resolv.conf private 1 shared 0 slave 0
(00.039039)      1:     Mounting unsupported @./var/lib/mysql (0)
(00.039043)      1:     Bind /var/lib/docker/volumes/da0a45864a7af547cbec0b3c1193fc1ba0b5f6a7da965820d011841c39ca3251/_data to ./var/lib/mysql
(00.039058)      1: 108:./var/lib/mysql private 1 shared 0 slave 0
(00.039068)      1:     Mounting sysfs @./sys (0)
(00.039092)      1: 94:./sys private 1 shared 0 slave 0
(00.039098)      1:     Mounting tmpfs @./sys/fs/cgroup (0)
(00.039124)      1: 95:./sys/fs/cgroup private 1 shared 0 slave 0
(00.042435)      1: Found fd 1 (id pipe:[173136]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.042517)      1: Inherit fd 1 moved to 4 to resolve clash
(00.044695)      1:     Mounting cgroup @./sys/fs/cgroup/cpuset (0)
(00.044707)      1:     Bind /sys/fs/cgroup/cpuset to ./sys/fs/cgroup/cpuset
(00.044727)      1: 96:./sys/fs/cgroup/cpuset private 1 shared 0 slave 0
(00.044737)      1:     Mounting cgroup @./sys/fs/cgroup/systemd (0)
(00.044741)      1:     Bind /sys/fs/cgroup/systemd to ./sys/fs/cgroup/systemd
(00.044751)      1: 107:./sys/fs/cgroup/systemd private 1 shared 0 slave 0
(00.044758)      1:     Mounting cgroup @./sys/fs/cgroup/hugetlb (0)
(00.044762)      1:     Bind /sys/fs/cgroup/hugetlb to ./sys/fs/cgroup/hugetlb
(00.044772)      1: 106:./sys/fs/cgroup/hugetlb private 1 shared 0 slave 0
(00.044780)      1:     Mounting cgroup @./sys/fs/cgroup/net_prio (0)
(00.044784)      1:     Bind /sys/fs/cgroup/net_prio to ./sys/fs/cgroup/net_prio
(00.044793)      1: 105:./sys/fs/cgroup/net_prio private 1 shared 0 slave 0
(00.044800)      1:     Mounting cgroup @./sys/fs/cgroup/perf_event (0)
(00.044804)      1:     Bind /sys/fs/cgroup/perf_event to ./sys/fs/cgroup/perf_event
(00.044813)      1: 104:./sys/fs/cgroup/perf_event private 1 shared 0 slave 0
(00.044820)      1:     Mounting cgroup @./sys/fs/cgroup/blkio (0)
(00.044824)      1:     Bind /sys/fs/cgroup/blkio to ./sys/fs/cgroup/blkio
(00.044834)      1: 103:./sys/fs/cgroup/blkio private 1 shared 0 slave 0
(00.044841)      1:     Mounting cgroup @./sys/fs/cgroup/net_cls (0)
(00.044845)      1:     Bind /sys/fs/cgroup/net_cls to ./sys/fs/cgroup/net_cls
(00.044854)      1: 102:./sys/fs/cgroup/net_cls private 1 shared 0 slave 0
(00.044861)      1:     Mounting cgroup @./sys/fs/cgroup/freezer (0)
(00.044865)      1:     Bind /sys/fs/cgroup/freezer to ./sys/fs/cgroup/freezer
(00.044874)      1: 101:./sys/fs/cgroup/freezer private 1 shared 0 slave 0
(00.044881)      1:     Mounting cgroup @./sys/fs/cgroup/devices (0)
(00.044885)      1:     Bind /sys/fs/cgroup/devices to ./sys/fs/cgroup/devices
(00.044894)      1: 100:./sys/fs/cgroup/devices private 1 shared 0 slave 0
(00.044901)      1:     Mounting cgroup @./sys/fs/cgroup/memory (0)
(00.044905)      1:     Bind /sys/fs/cgroup/memory to ./sys/fs/cgroup/memory
(00.044914)      1: 99:./sys/fs/cgroup/memory private 1 shared 0 slave 0
(00.044921)      1:     Mounting cgroup @./sys/fs/cgroup/cpuacct (0)
(00.044935)      1:     Bind /sys/fs/cgroup/cpuacct to ./sys/fs/cgroup/cpuacct
(00.044945)      1: 98:./sys/fs/cgroup/cpuacct private 1 shared 0 slave 0
(00.044952)      1:     Mounting cgroup @./sys/fs/cgroup/cpu (0)
(00.044956)      1:     Bind /sys/fs/cgroup/cpu to ./sys/fs/cgroup/cpu
(00.044965)      1: 97:./sys/fs/cgroup/cpu private 1 shared 0 slave 0
(00.044973)      1:     Mounting tmpfs @./dev (0)
(00.044997)      1: 90:./dev private 1 shared 0 slave 0
(00.048570)      1: Found fd 1 (id pipe:[173136]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.048650)      1: Inherit fd 1 moved to 4 to resolve clash
(00.050842)      1:     Mounting devpts @./dev/pts (0)
(00.050888)      1: 91:./dev/pts private 1 shared 0 slave 0
(00.050898)      1:     Mounting mqueue @./dev/mqueue (0)
(00.050915)      1: 93:./dev/mqueue private 1 shared 0 slave 0
(00.050922)      1:     Mounting tmpfs @./dev/shm (0)
(00.050941)      1: 92:./dev/shm private 1 shared 0 slave 0
(00.054410)      1: Found fd 1 (id pipe:[173136]) in inherit fd list (caller inherit_fd_resolve_clash)
(00.054517)      1: Inherit fd 1 moved to 4 to resolve clash
(00.056708)      1:     Mounting tmpfs @./proc/timer_stats (0)
(00.056723)      1:     Bind ./dev/null to ./proc/timer_stats
(00.056750)      1: 60:./proc/timer_stats private 1 shared 0 slave 0
(00.056758)      1:     Mounting tmpfs @./proc/kcore (0)
(00.056763)      1:     Bind ./dev/null to ./proc/kcore
(00.056773)      1: 59:./proc/kcore private 1 shared 0 slave 0
(00.056780)      1: Move the root to .
(00.098318)      1: Mount procfs in crtools-proc.vIAihE
(00.107874)      1: Found fd 1 (id pipe:[173136]) in inherit fd list (caller close_old_fds)
(00.107896)      1: Found fd 2 (id pipe:[173137]) in inherit fd list (caller close_old_fds)
(00.107911)      1: Preparing info about shared resources
(00.107924)      1: Collecting 34/18 (flags 0)
(00.107954)      1: Collected [dev/null] ID 0x1
(00.107965)      1: Collected [var/lib/mysql/ibdata1] ID 0x4
(00.107970)      1: Collected [tmp/ib927rlu] ID 0x5
(00.107974)      1: Collected [tmp/ibO3tWnq] ID 0x6
(00.107978)      1: Collected [tmp/ibagarqm] ID 0x7
(00.107983)      1: Collected [tmp/ib3H3jxe] ID 0x8
(00.107987)      1: Collected [var/lib/mysql/ib_logfile0] ID 0x9
(00.107991)      1: Collected [var/lib/mysql/ib_logfile1] ID 0xa
(00.107996)      1: Collected [var/lib/mysql/ibtmp1] ID 0xb
(00.108000)      1: Collected [tmp/ib8mFcLa] ID 0xc
(00.108004)      1: Collected [var/lib/mysql/mysql/time_zone_name.ibd] ID 0xd
(00.108008)      1: Collected [var/lib/mysql/mysql/plugin.ibd] ID 0xe
(00.108013)      1: Collected [var/lib/mysql/mysql/innodb_table_stats.ibd] ID 0xf
(00.108017)      1: Collected [var/lib/mysql/mysql/innodb_index_stats.ibd] ID 0x10
(00.108021)      1: Collected [var/lib/mysql/mysql/gtid_executed.ibd] ID 0x11
(00.108026)      1: Collected [var/lib/mysql/mysql/time_zone.ibd] ID 0x14
(00.108030)      1: Collected [var/lib/mysql/mysql/time_zone_transition.ibd] ID 0x15
(00.108034)      1: Collected [var/lib/mysql/mysql/time_zone_transition_type.ibd] ID 0x16
(00.108039)      1: Collected [var/lib/mysql/mysql/time_zone_leap_second.ibd] ID 0x17
(00.108043)      1: Collected [var/lib/mysql/mysql/server_cost.ibd] ID 0x18
(00.108047)      1: Collected [var/lib/mysql/mysql/engine_cost.ibd] ID 0x19
(00.108052)      1: Collected [var/lib/mysql/mysql/user.MYI] ID 0x1a
(00.108056)      1: Collected [var/lib/mysql/mysql/user.MYD] ID 0x1b
(00.108060)      1: Collected [var/lib/mysql/mysql/db.MYI] ID 0x1c
(00.108065)      1: Collected [var/lib/mysql/mysql/db.MYD] ID 0x1d
(00.108069)      1: Collected [var/lib/mysql/mysql/proxies_priv.MYI] ID 0x1e
(00.108073)      1: Collected [var/lib/mysql/mysql/proxies_priv.MYD] ID 0x1f
(00.108077)      1: Collected [var/lib/mysql/mysql/tables_priv.MYI] ID 0x20
(00.108082)      1: Collected [var/lib/mysql/mysql/tables_priv.MYD] ID 0x21
(00.108086)      1: Collected [var/lib/mysql/mysql/columns_priv.MYI] ID 0x22
(00.108090)      1: Collected [var/lib/mysql/mysql/columns_priv.MYD] ID 0x23
(00.108095)      1: Collected [var/lib/mysql/mysql/procs_priv.MYI] ID 0x24
(00.108107)      1: Collected [var/lib/mysql/mysql/procs_priv.MYD] ID 0x25
(00.108112)      1: Collected [var/lib/mysql/mysql/servers.ibd] ID 0x26
(00.108116)      1: Collected [var/lib/mysql/mysql/event.MYI] ID 0x27
(00.108121)      1: Collected [var/lib/mysql/mysql/event.MYD] ID 0x28
(00.108125)      1: Collected [usr/sbin/mysqld] ID 0x29
(00.108129)      1: Collected [lib/x86_64-linux-gnu/libnss_files-2.13.so] ID 0x2a
(00.108134)      1: Collected [lib/x86_64-linux-gnu/libnss_nis-2.13.so] ID 0x2b
(00.108138)      1: Collected [lib/x86_64-linux-gnu/libnsl-2.13.so] ID 0x2c
(00.108142)      1: Collected [lib/x86_64-linux-gnu/libnss_compat-2.13.so] ID 0x2d
(00.108147)      1: Collected [lib/x86_64-linux-gnu/libc-2.13.so] ID 0x2e
(00.108151)      1: Collected [lib/x86_64-linux-gnu/libgcc_s.so.1] ID 0x2f
(00.108155)      1: Collected [lib/x86_64-linux-gnu/libm-2.13.so] ID 0x30
(00.108159)      1: Collected [usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17] ID 0x31
(00.108164)      1: Collected [lib/x86_64-linux-gnu/librt-2.13.so] ID 0x32
(00.108168)      1: Collected [lib/x86_64-linux-gnu/libz.so.1.2.7] ID 0x33
(00.108172)      1: Collected [lib/x86_64-linux-gnu/libdl-2.13.so] ID 0x34
(00.108176)      1: Collected [lib/x86_64-linux-gnu/libcrypt-2.13.so] ID 0x35
(00.108181)      1: Collected [lib/x86_64-linux-gnu/libaio.so.1.0.1] ID 0x36
(00.108185)      1: Collected [lib/x86_64-linux-gnu/libpthread-2.13.so] ID 0x37
(00.108189)      1: Collected [lib/x86_64-linux-gnu/ld-2.13.so] ID 0x38
(00.108193)      1: Collected [var/lib/mysql] ID 0x39
(00.108198)      1: Collected [.] ID 0x3a
(00.108203)      1:  `- ... done
(00.108206)      1: Collecting 47/53 (flags 0)
(00.108215)      1: Configuring remap 0x5 -> 0x1
(00.108219)      1: Opening ghost file 0x1 for tmp/ib927rlu
(00.108250)      1: Error (files-reg.c:109): Can't open ghost file //tmp/ib927rlu.cr.1.ghost: File exists
(00.108258)      1:  `- ... done
(00.118266) Error (cr-restore.c:1927): Restoring FAILED.

Any help? Thank you..

@klesgidisold
Copy link
Author

Forgot to say, host of checkpointed container os linux mint 17, kernel 1.19-vivid and the host of restored linux ubuntu 14.04, kernel 1.19.

@xemul
Copy link
Member

xemul commented Aug 10, 2015

(00.108250) 1: Error (files-reg.c:109): Can't open ghost file //tmp/ib927rlu.cr.1.ghost: File exists

Here's the error. The file /tmp/ib927lru.cr.1 exists, while shouldn't. Have you modified the filesystem somehow before restore?

@klesgidisold
Copy link
Author

What you mean if I have modified the fs before? I don't think so.. I only have applied the patches tha referred to the criu/docker page.. this file shouldn't have been there?

Do you think that if i remove it, it will be ok?

Thanks for the reply..

@xemul
Copy link
Member

xemul commented Aug 10, 2015

I mean -- what did you do to get the filesystem of your container on the destination node? The file I mentioned (ib927lru.cr.1.ghost) is created by CRIU upon restore with unique name, according to logs this file is created only once, but it's already there. I wonder how this could happen?

And yes, if you remove this file this particular error will go away.

@klesgidisold
Copy link
Author

@xemul thanks again.. You were right about this particular file. I run the docker restore command multiple times so probably the file was there from a previous execution on the same container. The right log file is the bellow:

...
...
...
...
(00.851903)      1:     pr1 Read page 7fffbd03b000 from self 7fffbd03b000/a8a1000
(00.851912)      1:     pr1 Read page 7fffbd03c000 from self 7fffbd03c000/a8a2000
(00.851920)      1:     pr1 Read page 7fffbd03d000 from self 7fffbd03d000/a8a3000
(00.851927)      1:     pr1 Read page 7fffbd03e000 from self 7fffbd03e000/a8a4000
(00.851934)      1:     pr1 Read page 7fffbd03f000 from self 7fffbd03f000/a8a5000
(00.851942)      1:     pr1 Read page 7fffbd040000 from self 7fffbd040000/a8a6000
(00.851949)      1:     pr1 Read page 7fffbd041000 from self 7fffbd041000/a8a7000
(00.851957)      1:     pr1 Read page 7fffbd042000 from self 7fffbd042000/a8a8000
(00.851965)      1:     pr1 Read page 7fffbd043000 from self 7fffbd043000/a8a9000
(00.851972)      1:     pr1 Read page 7fffbd044000 from self 7fffbd044000/a8aa000
(00.851979)      1:     pr1 Read page 7fffbd045000 from self 7fffbd045000/a8ab000
(00.851987)      1:     pr1 Read page 7fffbd046000 from self 7fffbd046000/a8ac000
(00.851995)      1:     pr1 Read page 7fffbd047000 from self 7fffbd047000/a8ad000
(00.852003)      1:     pr1 Read page 7fffbd048000 from self 7fffbd048000/a8ae000
(00.852010)      1:     pr1 Read page 7fffbd049000 from self 7fffbd049000/a8af000
(00.852018)      1:     pr1 Read page 7fffbd04a000 from self 7fffbd04a000/a8b0000
(00.852027)      1:     pr1 Read page 7fffbd105000 from self 7fffbd105000/a8b1000
(00.852035)      1:     pr1 Read page 7fffbd106000 from self 7fffbd106000/a8b2000
(00.852063)      1: nr_restored_pages: 43187
(00.852068)      1: nr_shared_pages:   0
(00.852071)      1: nr_droped_pages:   0
(00.853799)      1: Found fd 1 (id pipe:[22193]) in inherit fd list (caller close_old_fds)
(00.853816)      1: Found fd 2 (id pipe:[22194]) in inherit fd list (caller close_old_fds)
(00.853825)      1: cg: Move into 2
(00.853832)      1: cg:   `-> blkio//docker/011d301103ba100ff19ed6a5c1b7330d8d72fb1ee4428b1ef357265840a3bc1a/tasks
(00.853873)      1: cg:   `-> cpu//docker/011d301103ba100ff19ed6a5c1b7330d8d72fb1ee4428b1ef357265840a3bc1a/tasks
(00.853899)      1: cg:   `-> cpuacct//docker/011d301103ba100ff19ed6a5c1b7330d8d72fb1ee4428b1ef357265840a3bc1a/tasks
(00.853918)      1: cg:   `-> cpuset//docker/011d301103ba100ff19ed6a5c1b7330d8d72fb1ee4428b1ef357265840a3bc1a/tasks
(00.853954)      1: Error (cgroup.c:901): cg: Can't move into cpuset//docker/011d301103ba100ff19ed6a5c1b7330d8d72fb1ee4428b1ef357265840a3bc1a/tasks (-1/0): No space left on device
(00.868983) Error (cr-restore.c:1927): Restoring FAILED.

@xemul
Copy link
Member

xemul commented Aug 12, 2015

OK, now we have ENOSPC from cpuset.tasks write. This usually happens when the cpuset.cpus or .mems is not set. Can you check the restore logs for "Restored cgroup property value..." message?

@klesgidisold
Copy link
Author

I was not able to find the restored cgroup property value, but I may not be able to locate it. I uploaded the entire log file here https://gist.github.com/klesgidis/06eff8fc2eeb3b9b0dec for reference.
Thank you

@avagin
Copy link
Member

avagin commented Aug 12, 2015

Looks like we forgot to set the manage-cgroup option to restore a
containers.
On Aug 12, 2015 10:13, "Pavel Emelyanov" notifications@github.com wrote:

OK, now we have ENOSPC from cpuset.tasks write. This usually happens when
the cpuset.cpus or .mems is not set. Can you check the restore logs for
"Restored cgroup property value..." message?


Reply to this email directly or view it on GitHub
https://github.com/xemul/criu/issues/16#issuecomment-130194703.

@xemul
Copy link
Member

xemul commented Aug 12, 2015

Probably yes :) Asking for @boucher help again.

@boucher
Copy link

boucher commented Aug 12, 2015

@avagin In libcontainer? It seems like we're setting it:
https://github.com/opencontainers/runc/blob/master/libcontainer/container_linux.go#L499

@avagin
Copy link
Member

avagin commented Aug 12, 2015

@boucher you are right. So the question is still open.

@xemul
Copy link
Member

xemul commented Aug 12, 2015

I think I've found it :) @cyrillos , look at the prepare_cgroup_dirs(). If the faccessat() call succeeds (directory exists) we don't call for restore_special_cpuset_props().

@cyrillos
Copy link
Member

I'll take a look, thanks

@cyrillos
Copy link
Member

So, this kind of behavior (restoring cpuset props if only controller didn't exist) came from commit 003aed3 as far as I can say. Thus to figure out if it is indeed a reason one should setup --manage-cgroups=full instead of soft mode which is by-default.

@cyrillos
Copy link
Member

In commit 46e8aee we have added appropriate modes for RPC calls.

@cyrillos
Copy link
Member

@tych0 Tycho, do you remember the reason why in commit 003aed3 we started to restore special props iif they dont exist? Why dont we restore them unconditionally?

@tych0
Copy link
Contributor

tych0 commented Aug 12, 2015

Hi Cyrill,

On Wed, Aug 12, 2015 at 11:28:25AM -0700, Cyrill Gorcunov wrote:

@tych0 Tycho, do you remember the reason why in commit 003aed3 we started to restore special props iif they dont exist? Why dont we restore them unconditionally?

I think the discussion is here:

http://lists.openvz.org/pipermail/criu/2014-October/017084.html

Basically, if criu didn't create the cgroup, we try not to to touch
it.

Tycho

@cyrillos
Copy link
Member

Need to refresh memory ;) Thanks I'll try to read it tomorrow.

@xemul
Copy link
Member

xemul commented Aug 13, 2015

I think we just decided that if cgroup exists it should be properly configured. Right now it seems not to be the case :\ And the question from me -- if we take existing cgroup and this cgroup doesn't have cpuset.cpus/.mems set, where does this cgroup came from? Why created it and how?

@xemul xemul changed the title Docker - Restore container - criu failed: type NOTIFY errno 0 Pre-created cpuset cgroup with not configured .cpus/.mems causes restore to fail Aug 13, 2015
@xemul xemul added the bug label Aug 13, 2015
@avagin
Copy link
Member

avagin commented Sep 10, 2015

opencontainers/runc#253
I think this pull request should fix this problem.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants